Pith. sign in

REVIEW 2 major objections 5 minor 45 references

Coupling Planning with Episodic Memory in LLM Agents for Software Issue Resolution

T0 review · 2 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A repair agent whose plan guides memory recall and whose memory triggers replanning resolves 25 more of 500 issues than a matched baseline without the coupling.

desk verdict A solid, honestly-reported empirical systems paper showing that bidirectionally coupling planning and memory helps LLM repair agents; the headline gain is credible, but the RQ4 interaction claim is partially confounded by execution grounding. read the letter →

arxiv 2608.06811 v1 pith:UH342JTC submitted 2026-08-07 cs.SE cs.AI

Jiahao Zhang , Yifan Zhang , Yu Huang This is my paper · ORCID
classification cs.SEcs.AI
keywords LLMagentsautomatedsoftwareengineeringissueresolutionprogramrepairhierarchicalphaseplannerepisodicmemoryplan-memorycouplingexecutiongrounding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PMCoder is an LLM agent for fixing real software issues that treats a repair run as a long episode and couples two kinds of state: a hierarchical phase planner (exploration, hypothesis, implementation, verification, with backtracking) and an episodic memory of what the agent has read, tried, and observed. The paper's central claim is that coupling the two in both directions—the current phase shapes what memory retrieves, while memory-derived signals about repeated failures and file churn trigger replanning—resolves more issues than strengthening either component alone. On a 500-issue human-validated repair benchmark with a 30B open-weight coding model, the coupled agent resolves an average of 25 more cases than a harness-matched baseline, and a 2x2 ablation attributes a significant part of the gain to the plan–memory interaction rather than to the sum of the parts. The authors argue the mechanism is better state management: fewer repeated failed commands, fewer empty-patch give-ups, less context-window exhaustion, and more recovery from corrupted edits. A reader should care because long repair episodes are where current agents lose the thread, and this is a concrete way to hold the thread.

What carries the argument

The load-bearing object is the bidirectional plan–memory coupling, realized by four boundary crossings: phase→retrieval (phase selects the MMR budget, diversity weight, and graph weight), sub-task→retrieval (active sub-task keywords form the retrieval anchor), memory→plan (memory statistics trigger stuck detection and backtracking), and verdict→plan (issue-reproduction pass/fail verdicts harden verification-state transitions). Supporting machinery is a deterministic hierarchical phase planner that makes one LLM call per episode and detects phases per step with ordered rules over shell commands and reasoning text plus two-step hysteresis, an episodic memory whose nodes store message role, compressed content, and file-touch and edit metadata read from executed commands, and an injection channel that appends a marker-delimited plan-plus-memory block to the latest tool result rather than rewriting the message history. This machinery keeps the state delivery in-distribution for tool-calling models while making the memory retrieval and replanning decisions advisory rather than action-blocking.

What would settle it

Run the 2x2 component ablation with per-step phase labels randomized: the coupling claim predicts the +10.3-instance interaction term should shrink toward zero, because phase-conditioned retrieval budgets and memory-driven stuck signals would be aimed at the wrong states. A complementary check is to have human annotators label a random sample of steps and compare them with the rule-based phase detector; large disagreement would put the load-bearing premise in doubt.

Watch

Extended reading notes

Core claim

The paper discovers that a repair episode's plan and its memory should be designed as one mutually conditioning control structure. In PMCoder, the planner's phase and active sub-task set the retrieval budget, diversity pressure, and code-graph weights used by an MMR beam search over memory nodes, and memory-derived statistics (edit counts, read saturation, repeated normalized actions) feed stuck detection and backtracking. Verification-phase completion is gated on execution evidence: when a validated issue-reproduction script exists, the script is rerun after edits and a still-failing verdict blocks terminal verification. On the headline 500-issue benchmark the full system resolves 167.3/500 on average versus 142.3/500 for a baseline with planner, memory, and grounding disabled; the component ablation reports a +10.3-instance interaction (F(1,8)=10.92, p=0.011), meaning the coupled pair beats the additive expectation of the two components. The paper frames this as evidence that state loss is a distinct failure mode in repository-level repair, separate from the base model's reasoning limits.

Load-bearing premise

The load-bearing premise is that the rule-based phase detector labels each step's phase accurately enough that phase-conditioned retrieval and memory-derived stuck signals land on the right trajectory states; the paper reports no direct accuracy measurement for this detector.

Editorial extensions

If this is right

  • If the central claim is right, a substantial share of long-horizon repair failures are state-management failures, so agents can improve without changing the base model's reasoning ability.
  • Adding memory to a planner, or a planner to memory, yields more than the sum of the isolated components; the significant +10.3-instance interaction is the paper's core evidence for this complementarity.
  • The plan–memory substrate still helps when the execution-grounding gate never fires, since the unarmed-instance gain of +12 cases (+3.8pp) comes from the coupling alone.
  • Trajectory signatures move in the predicted direction: repeated failed commands drop to half, empty-patch exits to a third, context-window exhaustion to under half, and revert-then-refix recoveries rise by 46%.
  • The positive direction persists across other language models, a separate agent scaffold, and a terminal-task benchmark, which the authors read as evidence that the design is not tied to one model or harness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the authors leave implicit is that the same bidirectional coupling principle should transfer to other long-horizon agent tasks with distinguishable phases, such as scientific experimentation or system administration, not just software repair.
  • Because the rule-based phase detector is the only noisy link in the loop, improving its accuracy—or replacing it with a learned classifier—should enlarge the interaction effect; this is testable by measuring detector agreement against human labels.
  • The production substitution the authors hint at, using failing CI jobs or issue-reported reproduction steps instead of offline-extracted repro scripts, can be tested directly and would show how much of the armed-instance gain depends on script quality.
  • If the phase signal is poor, the coupling could degrade rather than help, so phase-detector accuracy is the key controllable variable to monitor when deploying this design.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes PMCoder, an LLM agent for repository-level issue resolution that couples a hierarchical phase planner (EXPLORATION/HYPOTHESIS/IMPLEMENTATION/VERIFICATION) with episodic memory whose retrieval is conditioned on the current plan phase, while memory-derived trajectory statistics feed stuck detection and replanning. Execution grounding via issue-reproduction verdicts hardens verification-phase completion. On SWE-bench Verified with Qwen3-Coder-30B, PMCoder resolves 167.3/500 versus 142.3/500 for a harness-matched baseline over three runs (+25.0 cases, +5.0pp, instance-level cluster-bootstrap CI [+14.3,+35.7], p<0.001). A 2x2 component ablation reports a significant plan-by-memory interaction (+10.3 instances, F(1,8)=10.92, p=0.011). Supporting probes show the same positive direction on DeepSeek-V4-Flash, Claude Haiku 4.5, an OpenHands port, and the TerminalWorld sample. Trajectory analyses show reduced failed-action recurrence, empty-patch exits, and context-window exhaustion.

Significance. The headline result is credible: it uses the official SWE-bench harness, three runs per arm, complete separation between the arms, and a conservative instance-level cluster bootstrap, and the hyperparameters were fixed before the final runs. The paper also provides an anonymized replication package. If the interaction evidence survives a clean ablation, the contribution is significant: it demonstrates a concrete bidirectional control structure between plan state and episodic memory, with an execution-grounded verification signal, in a realistic long-horizon repair setting. The RQ2 trajectory signatures are a useful process-level complement to output-level grading. The main weakness is that the RQ4 design does not independently vary execution grounding, so the interaction claim needs an additional experimental condition or a more restricted interpretation.

major comments (2)
  1. [Section V-D, with Section III-E and Section VII-B] The 2x2 ablation does not actually isolate the plan-by-memory interaction because execution grounding is a third factor that is structurally correlated with the planner. In the observed cells, plan-only and plan+memory include the verdict-to-plan completion predicate and edit-integrity recovery, while baseline and memory-only do not, because those mechanisms require an active planner. The reported interaction contrast Y11 - Y10 - Y01 + Y00 therefore equals the plan-by-memory interaction plus the memory-by-grounding interaction and associated three-way terms. The +10.3 interaction could be driven by memory being more valuable when repro verdicts gate completion, rather than by plan state conditioning retrieval. Section VII-B concedes that RQ4 does not separately estimate the contribution of execution grounding or edit-integrity recovery, but that concession understates the problem: the confound is in the main interaction contrast itself. Please add a design that varies grounding independently, such as memory-only+grounding and full-without-grounding cells, or explicitly reframe the claim as a 'plan-plus-grounding × memory' interaction and remove the implication that the interaction is purely plan-by-memory.
  2. [Section III-B and Section V-B] No accuracy or agreement measurement is reported for the rule-based phase detector, despite this detector being the 'plan' signal that conditions memory retrieval in the plan→memory direction. The paper states that the detector is 'cheap but noisy' and uses two-step hysteresis, but it does not quantify how often phases are correct, nor how errors propagate to the phase-conditioned retrieval budgets (Section III-C) or to memory-derived stuck signals. Without such a measurement, the RQ2 mechanism signatures in Table V and the django-13516 narrative cannot be attributed to correct phase-conditioned retrieval rather than to other differences between the arms. Please add either a hand-labeled accuracy evaluation on a sample of trajectories, or an ablation that replaces detected phases with oracle or random phases, to establish that the phase signal is informative.
minor comments (5)
  1. [Section V-B-c] The armed-versus-unarmed comparison is presented as showing the value of execution grounding, but armed and unarmed instances likely differ in issue characteristics beyond the availability of a reproduction script, and the unarmed analysis is based on a single paired run with no confidence interval; please soften the causal reading or provide additional uncertainty quantification.
  2. [Section IV-G and Section V-D] The repeated-run protocol treats the three runs per cell as exchangeable, but it is not stated whether the runs across cells share seeds or are paired; if the same seeds are reused across configurations, a repeated-measures formulation of the factorial test would be more appropriate.
  3. [Section III-B-a] The ordered rules for the phase detector are described qualitatively (file-inspection, diagnostic commands, file-mutating actions, test execution); listing the exact rule order or pointing to the replication package's implementation would improve reproducibility.
  4. [Section VII-A] The sentence claiming that stronger API models for script extraction 'would likely improve script quality rather than advantage the local Qwen agent' is speculative; it would be better phrased as an acknowledged limitation of the current construction.
  5. [Abstract and Section V-C-a] The phrase 'with at least 14 additional resolved cases (+2.8pp)' is slightly awkward because the two cross-model gains are +16 and +14; please state the per-model numbers explicitly in the abstract or use a clearer aggregate phrasing.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor mechanism-signature circularity; headline benchmark result is externally grounded.

  1. other [Section III-B(c) (Backtracking and replanning) vs. Section V-B(a), Table V]
    "the planner declares the agent stuck on any of several signals: persistently nonzero return codes, a single file edited too many times, file reads saturated without edits, or one normalized action repeated past a threshold ... Failed-action recurrence rate 0.0137 0.0069 0.50×"

    The RQ2 mechanism signature 'failed-action recurrence rate' counts re-issued normalized commands after failure, which is precisely one of the deterministic planner signals ('one normalized action repeated past a threshold'). Because the planner interrupts this pattern by marking the active sub-task failed and pushing recovery sub-tasks, the reported 0.50× reduction in repeated failed actions is to a substantial degree an architectural consequence of the stuck detector itself, not an independent measurement of plan–memory coupling. The paper presents this signature as trajectory evidence explaining the gains, but the direction of this particular signature is forced by the design input.

full rationale

PMCoder's headline claim is evaluated by the official SWE-bench harness, an external oracle that never reads the planner state, memory, injection blocks, or reproduction verdicts, so the +25.0 resolved-instance gain is not a fitted or self-referential outcome. The paper also states that thresholds and budgets were fixed from pilots before the final Verified-500 runs, not tuned on the final resolved counts. The RQ4 plan–memory interaction may be confounded with execution grounding because the ablation toggles only planning and memory while grounding remains active in planner-containing cells; the paper itself concedes this in Section VII-B ('does not separately estimate the contribution of execution grounding or edit-integrity recovery'). That is an internal-validity limitation, not a circular reduction, and the unarmed-strata analysis provides some independent support for the substrate without grounding. The self-citation to the structurally aligned subtask-level memory work is used only for novelty positioning, not as load-bearing evidence for the measured gains. The only notable circular element is the RQ2 failed-action recurrence signature, whose reduction is architecturally expected from the planner's stuck-detection rule; this is a secondary mechanism claim and does not undermine the externally graded headline result.

Assumptions & free parameters 11 free parameters · 6 assumptions · 0 invented entities

Everything in the ledger tracks assumptions that the paper relies on but does not derive: the benchmark as ground truth, the reliability of the phase detector, the validity of validated reproduction scripts, and the fairness of the matched baseline. The hyperparameters listed are pilot-fixed, not fitted to the final outcome metric, but they are still choices made by the authors rather than derived values.

free parameters (11)
  • MMR lambda = 0.7
    Relevance-diversity tradeoff in retrieval scoring (Table II).
  • Lexical and graph weights (w_c, w_g) = 0.5 / 0.5
    Base relevance mix in memory scoring (Table II).
  • Default memory budget = 16k tokens
    Retrieval budget when no phase policy is active (Table II).
  • Memory node cap = 4k chars
    Per-node size cap for stored observations (Table II).
  • Phase budgets = 20k/16k/12k/16k/20k for exploration/hypothesis/implementation/verification/backtrack
    Phase-conditioned retrieval budgets (Table II).
  • Phase hysteresis = 2 steps
    Debounce for backward or lateral phase changes (Table II).
  • Minimum backtrack step = 10
    Suppresses early recovery triggers (Table II).
  • Backtrack cooldown = 8 steps
    Rate-limits replanning (Table II).
  • Goal-stack depth = 16
    Bounds recovery stack (Table II).
  • Stuck-detection thresholds = failed returns 5, repeated edits 6, read saturation 15, repeated action 8
    Signals that trigger replanning (Table II).
  • Repro check cap and timeout = 4 checks per episode, 180s timeout
    Bounds execution-grounding cost (Table II).
assumptions (6)
  • domain assumption SWE-bench Verified and its official Docker harness provide a valid, externally graded resolution signal.
    The headline metric is the official harness pass/fail; the paper does not derive this proxy (Section IV-A).
  • domain assumption FP8 serving at temperature 0 is run-stochastic, so repeated runs can be treated as replicates.
    This stated property motivates the three-run protocol (Section IV-G).
  • domain assumption The rule-based phase detector maps shell commands and reasoning text to the correct phase often enough for the coupling to help.
    The detector is the sole source of phase labels that condition memory retrieval and completion (Section III-B-b); accuracy is not measured.
  • domain assumption Validated issue-reproduction scripts provide reliable pass/fail evidence about whether the bug still reproduces after an edit.
    Scripts are extracted by the same Qwen model and validated only to fail pre-edit (Section IV-F); residual errors are acknowledged in Section VII-A.
  • domain assumption Memory statistics computed by the episodic memory are faithful to actually executed commands, not to model self-reports.
    Node metadata is read from the preceding executed command (Section III-C-a).
  • domain assumption The harness-matched baseline with all PMCoder components disabled is a fair control for isolating PMCoder's design.
    Baseline construction is described as the mini-SWE-agent loop with components disabled (Section IV-C).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Coupling Planning with Episodic Memory in LLM Agents for Software Issue Resolution." pith.science (2026). https://pith.science/paper/UH342JTC

@misc{pith2026260806811,
  author       = {Pith},
  title        = {Pith review of: Coupling Planning with Episodic Memory in LLM Agents for Software Issue Resolution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UH342JTC}},
  note         = {Machine review of arXiv:2608.06811}
}
abstract

Resolving a real software issue with a large language model (LLM) agent is a long repair episode, often tens to hundreds of steps spanning exploration, hypothesis, implementation, and verification. Success depends on both the base model's local reasoning and the agent's ability to maintain an evolving plan and remember observations across phases. Existing repository-level agents typically strengthen planning or memory in isolation, leaving long trajectories vulnerable to stale evidence, repeated failed edits, and verification inferred from the agent's own claims instead of execution evidence. We present PMCoder, an issue-resolution agent that couples a hierarchical phase planner with episodic memory. The coupling is bidirectional: the current plan phase conditions memory retrieval, while memory-derived trajectory statistics inform stuck detection and replanning. When available, issue-reproduction verdicts ground verification progress in execution evidence rather than self-reported completion. On SWE-bench Verified, PMCoder resolves an average of $25$ more cases ($+5.0$pp) than a harness-matched baseline, with gains persisting even where the reproduction gate never fires. Further Verified-500 evaluations show the same positive direction across Claude Haiku 4.5, DeepSeek-V4-Flash, and an OpenHands port, with at least $14$ additional resolved cases ($+2.8$pp). Separately, evaluation on TerminalWorld's official sample suggests that the plan-memory substrate transfers beyond issue reports. Ablation and trajectory analyses show where the gains come from: coupling planning and memory outperforms either component alone and reduces repeated failed actions, empty-patch exits, and context-window exhaustion.

Figures

Figures reproduced from arXiv: 2608.06811 by the authors.

Figure 1
Figure 1. A django-13516 trajectory illustrating plan–memory coupling: both agents find the same diagnosis, but PMCoder preserves and reuses it after edit recovery while the baseline drifts to the wrong file. right phase, and repeated failed work can change the plan rather than accumulate; we return to this trajectory in Sec￾tion V-B. This coupling relies on a shared notion of progress: the planner must know when to advance, … view at source ↗
Figure 2
Figure 2. PMCoder couples episodic memory with hierarchical planning inside the observe–act loop. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The phase planner advances sub-tasks through four phases with [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: PMCoder delivers plan–memory state by appending a marker [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Plan–memory interaction on Verified-500 (three-run means); the [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 12 canonical work pages

  1. [1]

    Cognitive architectures for language agents,

    T. R. Sumers, S. Yao, K. Narasimhan, and T. L. Griffiths, “Cognitive architectures for language agents,”Transactions on Machine Learning Research (TMLR), 2024. [Online]. Available: https://arxiv.org/abs/2309.02427

    arXiv 2024

  2. [2]

    Agents in software engineering: Survey, landscape, and vision,

    Y . Wang, W. Zhong, Y . Huang, E. Shi, M. Yang, J. Chen, H. Li, Y . Ma, Q. Wang, and Z. Zheng, “Agents in software engineering: Survey, landscape, and vision,” 2024. [Online]. Available: https://arxiv.org/abs/2409.09030

    arXiv 2024

  3. [3]

    SWE-agent: Agent-computer interfaces enable automated software engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. R. Narasimhan, and O. Press, “SWE-agent: Agent-computer interfaces enable automated software engineering,” inThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [Online]. Available: https://arxiv.org/abs/2405.15793

    arXiv 2024

  4. [4]

    Reflexion: Language agents with verbal reinforcement learning,

    N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: Language agents with verbal reinforcement learning,” inAdvances in Neural Information Processing Systems 36 (NeurIPS 2023), 2023. [Online]. Available: https://arxiv.org/abs/2303. 11366

    work page 2023

  5. [5]

    Autocoderover: Autonomous program improvement,

    Y . Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “Autocoderover: Autonomous program improvement,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2024). Vienna, Austria: ACM, 2024, pp. 1592–1604. [Online]. Available: https://arxiv.org/abs/2404.05427

    arXiv 2024

  6. [6]

    Codeplan: Repository-level coding using llms and planning,

    R. Bairi, A. Sonwane, A. Kanade, V . D. C., A. Iyer, S. Parthasarathy, S. Rajamani, B. Ashok, and S. Shet, “Codeplan: Repository-level coding using llms and planning,”Proceedings of the ACM on Software Engineering, vol. 1, no. FSE, Jul. 2024. [Online]. Available: https://doi.org/10.1145/3643757

    doi:10.1145/3643757 2024

  7. [7]

    SWE-search: Enhancing software agents with monte carlo tree search and iterative refinement,

    A. Antoniades, A. ¨Orwall, K. Zhang, Y . Xie, A. Goyal, and W. Y . Wang, “SWE-search: Enhancing software agents with monte carlo tree search and iterative refinement,” inThe Thirteenth International Conference on Learning Representations (ICLR), 2025. [Online]. Available: https://openreview.net/forum?id=G7sIFXugTX

    work page 2025

  8. [8]

    Swe-exp: Experience-driven software issue resolution,

    S. Chen, S. Lin, Y . Shi, H. Lian, X. Gu, L. Yun, D. Chen, L. Cao, J. Liu, N. Xia, and Q. Wang, “Swe-exp: Experience-driven software issue resolution,” 2025. [Online]. Available: https://arxiv.org/abs/2507.23361

    arXiv 2025

Show all 45 references
  1. [9]

    EXPEREPAIR: Dual-memory enhanced llm-based repository-level program repair,

    F. Mu, J. Wang, L. Shi, S. Wang, S. Li, and Q. Wang, “EXPEREPAIR: Dual-memory enhanced llm-based repository-level program repair,” 2026, accepted by FSE 2026. [Online]. Available: https://arxiv.org/abs/2506.10484

    2026 arXiv

  2. [10]

    Confucius code agent: Scalable agent scaffolding for real-world codebases,

    S. Wong, Z. Qi, Z. Wang, N. Hu, S. Lin, J. Ge, E. Gao, W. Chen, Y . Du, M. Yu, and Y . Zhang, “Confucius code agent: Scalable agent scaffolding for real-world codebases,” 2025. [Online]. Available: https://arxiv.org/abs/2512.10398

    2025

  3. [11]

    The extended mind,

    A. Clark and D. J. Chalmers, “The extended mind,”Analysis, vol. 58, no. 1, pp. 7–19, 1998. [Online]. Available: https://doi.org/10.1093/ analys/58.1.7

    1998

  4. [12]

    Cognitive offloading,

    E. F. Risko and S. J. Gilbert, “Cognitive offloading,”Trends in Cognitive Sciences, vol. 20, no. 9, pp. 676–688, 2016

    2016

  5. [13]

    Hutchins,Cognition in the Wild

    E. Hutchins,Cognition in the Wild. Cambridge, MA: MIT Press, 1995

    1995

  6. [14]

    RAP: Retrieval-augmented planning with contextual memory for multimodal LLM agents,

    T. Kagaya, T. J. Yuan, Y . Lou, J. Karlekar, S. Pranata, A. Kinose, K. Oguri, F. Wick, and Y . You, “RAP: Retrieval-augmented planning with contextual memory for multimodal LLM agents,” 2024. [Online]. Available: https://arxiv.org/abs/2402.03610

    2024 arXiv

  7. [15]

    Structurally aligned subtask-level memory for software engineering agents,

    K. Shen, J. Zhang, C. Sun, W. Zeng, and Y . Yue, “Structurally aligned subtask-level memory for software engineering agents,” 2026. [Online]. Available: https://arxiv.org/abs/2602.21611

    2026

  8. [16]

    The use of MMR, diversity-based reranking for reordering documents and producing summaries,

    J. Carbonell and J. Goldstein, “The use of MMR, diversity-based reranking for reordering documents and producing summaries,” in Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’98). New York, NY , US...

    1998

  9. [17]

    Large language models cannot self- correct reasoning yet,

    J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. W. Yu, X. Song, and D. Zhou, “Large language models cannot self- correct reasoning yet,” inThe Twelfth International Conference on Learning Representations (ICLR), 2024. [Online]. Available: https://openreview.net/forum?id=IkmD3fKBPQ

    2024

  10. [18]

    CRITIC: Large language models can self-correct with tool-interactive critiquing,

    Z. Gou, Z. Shao, Y . Gong, Y . Shen, Y . Yang, N. Duan, and W. Chen, “CRITIC: Large language models can self-correct with tool-interactive critiquing,” inThe Twelfth International Conference on Learning Representations (ICLR), 2024. [Online]. Available: https://arxiv.org/abs/2...

    2024 arXiv

  11. [19]

    SWE-bench: Can language models resolve real-world Github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “SWE-bench: Can language models resolve real-world Github issues?” inThe Twelfth International Conference on Learning Representations (ICLR), 2024. [Online]. Available: https://openreview.net/fo...

    2024

  12. [20]

    Introducing SWE-bench Verified,

    OpenAI, “Introducing SWE-bench Verified,” OpenAI, 2024. [Online]. Available: https://openai.com/index/introducing-swe-bench-verified/

    2024

  13. [21]

    OpenHands: An open platform for AI software developers as generalist agents,

    X. Wang, B. Li, Y . Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y . Song, B. Li, J. Singh, H. H. Tran, F. Li, R. Ma, M. Zheng, B. Qian, Y . Shao, N. Muennighoff, Y . Zhang, B. Hui, J. Lin, R. Brennan, H. Peng, H. Ji, and G. Neubig, “OpenHands: An open platform for AI software d...

    2025

  14. [22]

    ADaPT: As-needed decomposition and planning with language models,

    A. Prasad, A. Koller, M. Hartmann, P. Clark, A. Sabharwal, M. Bansal, and T. Khot, “ADaPT: As-needed decomposition and planning with language models,” inFindings of the Association for Computational Linguistics: NAACL 2024, K. Duh, H. Gomez, and S. Bethard, Eds. Mexico City, M...

    2024

  15. [23]

    An analysis of patch plausibility and correctness for generate-and-validate patch generation systems,

    Z. Qi, F. Long, S. Achour, and M. Rinard, “An analysis of patch plausibility and correctness for generate-and-validate patch generation systems,” inProceedings of the 2015 International Symposium on Software Testing and Analysis (ISSTA 2015). Baltimore, MD, USA: ACM, 2015, pp....

    2015

  16. [24]

    Is the cure worse than the disease? overfitting in automated program repair,

    E. K. Smith, E. T. Barr, C. Le Goues, and Y . Brun, “Is the cure worse than the disease? overfitting in automated program repair,” in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2015). Bergamo, Italy: ACM, 2015, pp. 532–543. [Onl...

    2015

  17. [25]

    The effects of reward misspecification: Mapping and mitigating misaligned models,

    A. Pan, K. Bhatia, and J. Steinhardt, “The effects of reward misspecification: Mapping and mitigating misaligned models,” in The Tenth International Conference on Learning Representations (ICLR), 2022. [Online]. Available: https://openreview.net/forum?id= JYtwGwIL7ye

    2022

  18. [26]

    Defining and characterizing reward hacking,

    J. Skalse, N. H. R. Howe, D. Krasheninnikov, and D. Krueger, “Defining and characterizing reward hacking,” inAdvances in Neural Information Processing Systems 35 (NeurIPS 2022), 2022. [Online]. Available: https://arxiv.org/abs/2209.13085

    2022 arXiv

  19. [27]

    Sycophancy to subterfuge: Investigating reward-tampering in large language models,

    C. Denison, M. MacDiarmid, F. Barez, D. Duvenaud, S. Kravec, S. Marks, N. Schiefer, R. Soklaski, A. Tamkin, J. Kaplan, B. Shlegeris, S. R. Bowman, E. Perez, and E. Hubinger, “Sycophancy to subterfuge: Investigating reward-tampering in large language models,”arXiv preprint arXi...

    2024 arXiv

  20. [28]

    Agentic uncertainty reveals agentic overconfidence,

    J. Kaddour, S. Patel, G. Dovonon, L. Richter, P. Minervini, and M. J. Kusner, “Agentic uncertainty reveals agentic overconfidence,” 2026. [Online]. Available: https://arxiv.org/abs/2602.06948

    2026

  21. [29]

    GenProg: A generic method for automatic software repair,

    C. Le Goues, T. Nguyen, S. Forrest, and W. Weimer, “GenProg: A generic method for automatic software repair,”IEEE Transactions on Software Engineering, vol. 38, no. 1, pp. 54–72, 2012. [Online]. Available: https://doi.org/10.1109/TSE.2011.104

    2012 doi

  22. [30]

    HiAgent: Hierarchical working memory management for solving long-horizon agent tasks with large language model,

    M. Hu, T. Chen, Q. Chen, Y . Mu, W. Shao, and P. Luo, “HiAgent: Hierarchical working memory management for solving long-horizon agent tasks with large language model,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap...

    2025

  23. [31]

    Simplifying and isolating failure-inducing input,

    A. Zeller and R. Hildebrandt, “Simplifying and isolating failure-inducing input,”IEEE Transactions on Software Engineering, vol. 28, no. 2, pp. 183–200, 2002. [Online]. Available: https://doi.org/10.1109/32.988498

    2002 doi

  24. [32]

    Qwen3 technical report,

    Q. Team, “Qwen3 technical report,” 2025. [Online]. Available: https://arxiv.org/abs/2505.09388

    2025 arXiv

  25. [33]

    Efficient memory management for large language model serving with PagedAttention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with PagedAttention,” inProceedings of the 29th Symposium on Operating Systems Principles (SOSP). ACM, 2023, pp. 611...

    2023 arXiv

  26. [34]

    TerminalWorld: Benchmarking agents on real-world terminal tasks,

    Z. Chu, J. Hu, X. Jiang, P. Zou, H. Li, C. Peng, P. O’Hearn, E. T. Barr, M. Harman, F. Sarro, and H. Ye, “TerminalWorld: Benchmarking agents on real-world terminal tasks,” 2026. [Online]. Available: https://arxiv.org/abs/2605.22535

    2026 arXiv

  27. [35]

    Agentless: Demystifying LLM-based software engineering agents,

    C. S. Xia, Y . Deng, S. Dunn, and L. Zhang, “Agentless: Demystifying LLM-based software engineering agents,”Proceedings of the ACM on Software Engineering, vol. 2, no. FSE, pp. 801–824, 2025, proceedings of the 33rd ACM SIGSOFT International Symposium on the Foundations of Sof...

    2025 arXiv

  28. [36]

    Specrover: Code intent extraction via llms,

    H. Ruan, Y . Zhang, and A. Roychoudhury, “Specrover: Code intent extraction via llms,” in2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), 2025, pp. 963–974. [Online]. Available: https://arxiv.org/abs/2408.02232

    2025 arXiv

  29. [37]

    Identifying patch correctness in test-based program repair,

    Y . Xiong, X. Liu, M. Zeng, L. Zhang, and G. Huang, “Identifying patch correctness in test-based program repair,” inProceedings of the 40th International Conference on Software Engineering (ICSE). Gothenburg, Sweden: ACM, 2018, pp. 789–799. [Online]. Available: https://doi.org...

    2018

  30. [38]

    Plan-and-solve prompting: Improving zero-shot chain- of-thought reasoning by large language models,

    L. Wang, W. Xu, Y . Lan, Z. Hu, Y . Lan, R. K.-W. Lee, and E.-P. Lim, “Plan-and-solve prompting: Improving zero-shot chain- of-thought reasoning by large language models,” inProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long...

    2023

  31. [39]

    Adaplanner: Adaptive planning from feedback with language models,

    H. Sun, Y . Zhuang, L. Kong, B. Dai, and C. Zhang, “Adaplanner: Adaptive planning from feedback with language models,” 2023. [Online]. Available: https://arxiv.org/abs/2305.16653

    2023 arXiv

  32. [40]

    Generative agents: Interactive simulacra of human behavior,

    J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein, “Generative agents: Interactive simulacra of human behavior,” inProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, ser. UIST ’23. New York, NY , USA: Associatio...

    2023

  33. [41]

    MemGPT: Towards LLMs as operating systems,

    C. Packer, S. Wooders, K. Lin, V . Fang, S. G. Patil, I. Stoica, and J. E. Gonzalez, “MemGPT: Towards LLMs as operating systems,”arXiv preprint arXiv:2310.08560, 2023. [Online]. Available: https://arxiv.org/abs/2310.08560

    2023 arXiv

  34. [42]

    V oyager: An open-ended embodied agent with large language models,

    G. Wang, Y . Xie, Y . Jiang, A. Mandlekar, C. Xiao, Y . Zhu, L. Fan, and A. Anandkumar, “V oyager: An open-ended embodied agent with large language models,”Transactions on Machine Learning Research (TMLR), 2024. [Online]. Available: https://arxiv.org/abs/2305.16291

    2024 arXiv

  35. [43]

    A- MEM: Agentic memory for LLM agents,

    W. Xu, Z. Liang, K. Mei, H. Gao, J. Tan, and Y . Zhang, “A- MEM: Agentic memory for LLM agents,” inAdvances in Neural Information Processing Systems (NeurIPS), 2025. [Online]. Available: https://arxiv.org/abs/2502.12110

    2025 arXiv

  36. [44]

    ReasoningBank: Scaling agent self-evolving with reasoning memory,

    S. Ouyang, J. Yan, I.-H. Hsu, Y . Chen, K. Jiang, Z. Wang, R. Han, L. T. Le, S. Daruki, X. Tang, V . Tirumalashetty, G. Lee, M. Rofouei, H. Lin, J. Han, C.-Y . Lee, and T. Pfister, “ReasoningBank: Scaling agent self-evolving with reasoning memory,” inInternational Conference o...

    2026 arXiv

  37. [45]

    Large language models are few-shot testers: Exploring LLM-based general bug reproduction,

    S. Kang, J. Yoon, and S. Yoo, “Large language models are few-shot testers: Exploring LLM-based general bug reproduction,” inProceedings of the 45th International Conference on Software Engineering (ICSE), ser. ICSE ’23. IEEE Press, 2023, pp. 2312–2323. [Online]. Available: htt...

    2023 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.