Pith.

Software Engineering · arXiv:2611.03219 v1 · submitted 2 Nov 2026 · 51 references

Can Coding Agents Sustain Continual Evolution without Forgetting?

Shangziqi Zhao, Yitong Zhang, Jia Li

The paper says

“Recently, harness self-evolution has emerged as an important paradigm for improving agent performance. Prior studies show that evolving the agent harness can effectively enhance agent capabilities without modifying the foundation model's parameters.”

more of the abstract

We find that existing approaches largely study agent evolution on a single task. In practice, coding agents may need to handle multiple task types over time. Whether an agent harness can continuously evolve across multiple tasks without suffering from catastrophic forgetting remains an important open question. In this paper, we investigate this open question through a comprehensive empirical study of continual harness self-evolution. We organize three software engineering task datasets into six different evolution directions and apply two representative harness evolution methods to perform continual evolution along these directions. Our experimental results show that evolution on a subsequent dataset can substantially undermine the performance on previously seen datasets. For example, with DeepSeek-V4-Flash-0731, Meta-Harness improves its performance on SWE-Bench to 94.0% after evolution. However, after further evolution on FeatureBench, its performance on SWE-Bench drops to 32.0% by the fifth round, corresponding to a 66% relative decrease. Finally, based on our experimental findings, we identify several insights, providing directions for the continual self-evolution of agent harnesses. Code is available at https://github.com/THU-Agent/continual-harness-evolution.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 3 Nov 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. First real evidence that harness self-evolution suffers catastrophic forgetting across task types, but the missing held-out evaluation split leaves the headline magnitudes uncalibrated. This line is wrong

3 major objections · 4 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 51 references say.

The paper says, at Sections 3.1 and 4.1; Eqs. (4) and (5); Appendix B

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

The headline forgetting numbers are measured on the same instances used for self-evolution. Section 3.1 states that 50 SWE-Bench, 50 SWT-Bench, and 30 FeatureBench instances were sampled 'for self-evolution', and Section 4.1 selects the best harness from single-task evolution and then computes the drops with Eqs.

more of objection 1

(4) and (5) on that same source sample. No held-out evaluation split is introduced anywhere. The 94.0 to 32.0 SWE-Bench drop and the 100%/98% degradations are therefore drops on the exact tasks the harness was optimized on, not on unseen instances of the same task type. Appendix B even documents that evolved modifications can be surface-form-specific, e.g. the _TEST_DELIVERABLE_HINTS branch keyed to the SWT-Bench prompt template. The observed degradation is thus compatible both with true loss of a transferable capability and with overwriting of instance- or template-specific overfitting. Because the abstract's practical motivation and the phrase 'previously acquired capabilities' require the general-capability reading, the central RQ1 claim needs a held-out split, with forgetting reported separately on the evolution set and on unseen instances.

The paper says, at Section 4.2.2; Table 1

The review did not quote this passage.

Pith says: Objection 2 of 3 · major

The verification-conflict ablation is drawn from a different evolution direction than the setting it is used to explain. The authors state that SWE-Bench and FeatureBench do not differ substantially in submission artifacts, so they take the verification-conflict modification from the SWE-to-SWT direction instead.

more of objection 2

Consequently Table 1 demonstrates that a mandatory test-only artifact gate can destroy SWE-Bench performance, but it does not establish that verification conflicts caused the observed SWE-to-Feature degradation. This is a load-bearing gap for RQ2's causal attribution. The paper should either supply a verification-conflict modification extracted from the SWE-to-Feature trajectories, or re-frame the RQ2 result as showing potential mechanisms rather than established causes of the observed drops.

The paper says, at Section 4.2.1

The review did not quote this passage.

Pith says: Objection 3 of 3 · major

The conflict taxonomy and the reported proportions (87.1%, 8.2%, 0.6%) are produced by manual inspection of the same trajectories that exhibited pass-to-fail transitions, with no coding protocol, no inter-annotator agreement measure, and no statement of how the 520 trajectories were allocated across the six directions.

more of objection 3

This is acceptable as hypothesis generation, and the Table 1 ablation is a useful independent check, but the current presentation labels the result as root-cause analysis. Adding a short coding protocol and agreement statistic, or at minimum re-labeling the analysis as an exploratory categorization, would make the strength of the RQ2 evidence match the claims.

4 minor objections

The paper says, at Section 4.1 and Figure 3

Pith says: minor objection 1 of 4

All results come from a single sampled subset and a single run with no error bars or repeated evaluations. The large effect sizes make the basic RQ1 existence claim robust to sampling noise, but the magnitudes (29%, 66%, 98%, 100%) should be presented as point estimates from one run rather than as calibrated measurements.

The paper says, at Section 4.1, paragraph on generalization

Pith says: minor objection 2 of 4

The sentence claiming that forgetting on adjacent tasks 'will likewise cause forgetting in continual evolution across multiple datasets' is an extrapolation, not a demonstrated result. It is plausible, but it should be labeled as a conjecture or supported by at least one three-dataset sequence.

show 2 more

The paper says, at Table 1

Pith says: minor objection 3 of 4

The agent time column reports aggregated seconds without stating whether the numbers are means, medians, or totals, or over how many instances they are computed. This should be clarified for reproducibility.

The paper says, at Appendix B

Pith says: minor objection 4 of 4

The Appendix B case study is one of the strongest parts of the paper because it directly shows template-level overfitting in evolved harness code. The main text should point readers to this appendix early, since it motivates both the SWT paraphrasing decision and the need for held-out evaluation.

What would settle it

Rerun the SWE→Feature and SWE→SWT continual-evolution protocols with the same 50-instance evolution splits, but evaluate source-task performance at Initial, Iter 3, and Iter 5 on an independent held-out subset (for example, another 50 SWE-Bench Verified instances not used in evolution, or the full 500-instance set). If the held-out relative drops are comparable to the reported 29%, 66%, and 98–100% drops, the forgetting claim stands as a loss of general capability. If held-out performance stays near its pre-evolution level while the evolution-set score collapses, the headline numbers are at least partly an artifact of evaluating on the evolution set.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 3 Nov 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic partly agrees 1 min 7 s · 16,452 tokens read, 9,841 written
  3. Editor wrote the verdict above 45 s · 17,338 tokens read, 6,690 written
  4. Referee major revision: 3 major objections, 4 minor 41 s · 16,726 tokens read, 5,720 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The thing to know: this is the first demonstration I've seen that harness self-evolution, which looks great within a single task type, suffers severe catastrophic forgetting when evolution continues across task types. AHE and Meta-Harness push SWE-Bench to 94-96% on their own, then lose 66-100% of that after evolving on FeatureBench or SWT-Bench. That is a genuine warning for a research area whose papers mostly report single-benchmark gains, and it is not present in the cited prior work.

The paper earns its keep. The conflict taxonomy — verification/artifact gates, workflow changes, budget strategy — comes from manual analysis of 520 trajectories and is concrete, and Table 1's ablation gives it an independent handle: injecting a verification conflict drops SWE-Bench from 96 to 0. The Appendix B discussion of the _TEST_DELIVERABLE_HINTS condition shows the authors saw the surface-form overfitting failure mode and tried to mitigate it by paraphrasing SWT-Bench prompts. That is honest engagement.

The soft spots are real but not fatal. The main one: forgetting is measured on the same 50/50/30 instances used to evolve the harness, with no held-out split. So 94→32 could partly be loss of instance-specific overfitting rather than lost general capability. That criticism has teeth, but it does not fully land: the drops go well below the initial harness's 82% baseline, to 32% and even 0%. Overfitting loss alone would not do that — the evolved harness actively blocks previously solvable behavior, and the artifact-gate mechanism in RQ2 operates at task-type level, not instance level. So the phenomenon has a solid core; the headline magnitudes are just uncalibrated. Adding a held-out evaluation split is the one change I would require before fully trusting the quantitative claims, and it is feasible.

Smaller caveats, in proportion: one foundation model, no seeds or repeated runs, and the Table 1 ablation borrows its verification-conflict modification from a different evolution direction than the setting it studies. These affect the magnitudes, not the direction.

Anyone working on harness self-evolution or agent continual learning should read this. It deserves a serious referee: I would send it to review with a request for held-out evaluation and variance estimates, not a desk reject. The central finding is important enough that the fixable measurement gap is worth the referee cycle.

The strongest objection

Section 3.1 samples 50 SWE-Bench, 50 SWT-Bench, and 30 FeatureBench instances "for self-evolution"; Section 4.1 then selects the best-performing harness from single-task evolution and measures forgetting with Eq. (4)/(5) on the earlier dataset. The paper introduces no held-out evaluation split, so the headline result (94.0→32.0 on SWE-Bench) is a drop on the exact tasks the harness was optimized on. That is compatible with catastrophic forgetting, but it is also exactly what would be observed if single-task evolution inserted instance- or sample-specific heuristics and target-task evolution subsequently overwrote them; the Appendix's `_TEST_DELIVERABLE_HINTS` case shows the evolvers do produce surface-form-specific conditions. The practical motivation ("agents handle multiple task types over time") and the wording "previously acquired capabilities" require the phenomenon to hold on tasks not used to evolve the harness. As it stands, RQ1 conflates forgetting on the evolution set with loss of transferable capability, which affects the central claim, not only RQ2's mechanism attribution or variance estimates.

The reader's CONDITIONAL verdict is reasonable, but the weakest point is not the RQ2 ablation. The RQ1 evidence itself is measured on the evolution set: the 50/50/30 samples in Section 3.1 are used both to evolve the harness and, as Table 2 suggests, as the evaluation datasets, with no held-out split and no stated seeds. Because a harness is a small code-plus-prompt object, single-task evolution can readily overfit the sampled instances; the Appendix's template-hint example shows such sample-specific conditions actually occur. Once the harness is re-evolved for a different task type, those conditions disappear, producing large drops even if no general capability existed. The reader's RQ2 concern is real (representative modifications were selected after the drops), but it is secondary: even a perfect mechanistic story would not establish that the drops generalize beyond the evolution instances. I therefore propose a held-out evaluation as the single check that would settle whether the central "not transferable / destroys prior capabilities" claim holds. The study otherwise has independent support: two methods, six directions, large effect sizes, manual trajectory analysis of 520 trajectories, and a code release. Assuming the held-out check reproduces the drops, the current CONDITIONAL verdict stands; if it does not, the paper should be reframed as evidence of sample-level overfitting rather than cross-task forgetting.

The referee's report

If the central result survives evaluation on held-out instances, this is a valuable negative result for the harness self-evolution community: it identifies a concrete stability problem and gives the field a new evaluation protocol and a testable conflict taxonomy. The paper has real strengths: the two evolution methods are architecturally different yet show consistent large-magnitude drops; the conflict taxonomy is concrete and the Table 1 ablation provides an independent falsifiable handle; and the authors ship code and a detailed reproducibility configuration. The significance is limited by the absence of a held-out evaluation split, reliance on one foundation model and one sampled subset without repeated runs, and the post-hoc nature of the RQ2 taxonomy.

Pith's reading of the paper (a machine's summary, not the authors')

This paper asks whether a coding agent's harness, the prompts, control flow, and tools wrapped around a fixed foundation model, can be evolved continuously across different software engineering task types without destroying what was learned earlier. The paper reports that current self-evolution methods cannot: across six two-stage evolution directions spanning code repair, test generation, and feature implementation, evolving on the second task type substantially degraded performance on the first, in the worst case erasing 100% of prior SWE-Bench performance. It formalizes the problem with explicit retention conditions, quantifies forgetting as absolute and relative drops on earlier datasets, and explains the drops through a taxonomy of harness-level conflicts: verification, workflow, and budget strategy, derived from manual analysis of 520 execution trajectories. The practical point is that reported single-task evolution gains are not automatically general agent improvement, and deployed agents facing a sequence of task types may silently lose earlier capabilities.

What carries the argument. The central object is the agent harness H: the runtime layer of system prompts, execution logic, and tool definitions wrapped around a fixed foundation model M, with the agent written as A = ⟨M, H⟩ and an evolution step written H' = Ω(H, T), where Ω revises the harness from execution trajectories T collected on the current dataset. The paper's formal target is the pair of retention conditions, Eval(H_{i+1}, D_i) > Eval(H_i, D_i) and Eval(H_{i+1}, D_j) ≥ Eval(H_{j+1}, D_j) for j < i, with forgetting quantified as Δ_forget and ρ_forget, the absolute and relative drops on earlier datasets. The explanatory machinery is the conflict taxonomy, verification, workflow, and budget strategy, extracted by diffing harness code before and after continual evolution and tracing pass-to-fail transitions in 520 trajectories; this taxonomy converts a measured performance drop into a mechanism that future evolution methods could be designed to avoid.

Core claim. The paper's central claim is that catastrophic forgetting is real and large in harness self-evolution: a harness optimized for one task type and then re-optimized for a second type does not merely fail to transfer, it actively destroys capabilities built for the first type. With DeepSeek-V4-Flash-0731 as the fixed foundation model, Meta-Harness reaches 94.0% on SWE-Bench after evolution and then drops to 32.0% after five rounds of evolution on FeatureBench, a 66% relative decrease, while evolving from SWE-Bench to SWT-Bench costs AHE and Meta-Harness 100% and 98% of their SWE-Bench performance. The damage varies with direction and task relatedness: migrations between the two source-modifying benchmarks forget less than migrations involving the test-generation benchmark. The authors further argue the mechanism is identifiable: manual trajectory analysis attributes pass-to-fail transitions mostly to verification conflicts, at 87.1%, above all artifact conflicts in which the evolved harness enforces the wrong submission deliverable, followed by workflow conflicts at 8.2% and budget-strategy changes at 0.6%. They also note that the one mitigating pattern they observed, a conditional branch distinguishing task types, was keyed to surface prompt phrasing and would not generalize.

The premise it rests on. The RQ2 mechanism attribution assumes the few manually selected harness modifications used in the ablation faithfully reproduce the conflicts that drove forgetting in the full continual runs, even though the verification-conflict example was drawn from a different evolution direction than the setting studied and no error bars are reported.

If the paper is right (Pith's inferences, not the authors')

Single-task evolution results can mislead: the 94.0% SWE-Bench score achieved by isolated evolution is not a durable capability once the harness is further evolved toward a different task type, so benchmark reports should be read as task-specific rather than general agent gains.

The retention conditions the paper defines, improve on the current task and do not fall on earlier tasks, give future harness-evolution methods a concrete success criterion to enforce rather than hope for.

Task relatedness is a usable predictor of forgetting risk: migration between tasks with the same deliverable type, source-code modification in both SWE-Bench and FeatureBench, forgets less than migration into the test-writing task SWT-Bench.

The three-way conflict taxonomy identifies where forgetting enters, giving designers specific surfaces to guard: submission gates and artifact expectations, workflow orderings, and budget strategies.

Evolved harness conditions should be keyed to task semantics rather than surface phrasing, since the paper shows a template-based conditional branch helped only while all prompts shared a fixed sentence structure.

A direct remedy the paper does not test: turn the retention conditions into a hard gate during evolution by re-evaluating candidate harnesses on a retained sample of earlier tasks and rejecting updates that drop prior scores, an analogue of experience replay for harnesses.

The near-total losses in the SWE-to-SWT direction look like overfitting to the target benchmark's submission protocol; if that is the mechanism, regularizing evolution toward minimal diffs should shrink forgetting, which is testable with the released code.

The budget-strategy result implies that pass/fail scores understate the harm of continual evolution: some conflicts leave correctness untouched while inflating task time, up to 235%, so efficiency-aware evaluation would catch damage that score-only tables miss.

The taxonomy should transfer beyond software engineering: any harness that enforces deliverable formats or workflow orderings, for instance in agentic web or data-science settings, should exhibit the same conflict types when the task type changes, which can be checked by replaying the same two-stage design on those domains.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Recently, harness self-evolution has emerged as an important paradigm for improving agent performance. Prior studies show that evolving the agent harness can effectively enhance agent capabilities without modifying the foundation model's parameters. We find that existing approaches largely study agent evolution on a single task. In practice, coding agents may need to handle multiple task types over time. Whether an agent harness can continuously evolve across multiple tasks without suffering from catastrophic forgetting remains an important open question. In this paper, we investigate this open question through a comprehensive empirical study of continual harness self-evolution. We organize three software engineering task datasets into six different evolution directions and apply two representative harness evolution methods to perform continual evolution along these directions. Our experimental results show that evolution on a subsequent dataset can substantially undermine the performance on previously seen datasets. For example, with DeepSeek-V4-Flash-0731, Meta-Harness improves its performance on SWE-Bench to 94.0% after evolution. However, after further evolution on FeatureBench, its performance on SWE-Bench drops to 32.0% by the fifth round, corresponding to a 66% relative decrease. Finally, based on our experimental findings, we identify several insights, providing directions for the continual self-evolution of agent harnesses. Code is available at https://github.com/THU-Agent/continual-harness-evolution.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-04

The authors' main claim, as the classifier picked it out 1.00

“Our experimental results show that evolution on a subsequent dataset can substantially undermine the performance on previously seen datasets.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.09 · states limitation 0.37 · human participants 0.03 · controlled comparison 0.34 · randomized assignment 0.02 · states quantitative result 0.98 · compares with prior results 0.67 · mentions code or data availability 0.98

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Recently, harness self-evolution has emerged as an important paradigm for improving agent performance. background 1.00 0.05 0.02 asserted 1.00
Prior studies show that evolving the agent harness can effectively enhance agent capabilities without modifying the foundation model's parameters. background 1.00 0.03 0.02 asserted 1.00
We find that existing approaches largely study agent evolution on a single task. gap 0.87 0.55 0.02 asserted 1.00
In practice, coding agents may need to handle multiple task types over time. background 0.81 0.09 0.02 hedged possibility 1.00
Whether an agent harness can continuously evolve across multiple tasks without suffering from catastrophic forgetting remains an important open question. gap 1.00 0.07 0.04 asserted 0.84
In this paper, we investigate this open question through a comprehensive empirical study of continual harness self-evolution. contribution 1.00 0.84 0.03 asserted 0.91
We organize three software engineering task datasets into six different evolution directions and apply two representative harness evolution methods to perform continual evolution along these directions. method 0.98 0.95 0.03 asserted 0.96
Our experimental results show that evolution on a subsequent dataset can substantially undermine the performance on previously seen datasets. result 1.00 0.95 0.78 asserted 1.00
For example, with DeepSeek-V4-Flash-0731, Meta-Harness improves its performance on SWE-Bench to 94.0% after evolution. result 1.00 0.87 0.04 asserted 1.00
However, after further evolution on FeatureBench, its performance on SWE-Bench drops to 32.0% by the fifth round, corresponding to a 66% relative decrease. result 1.00 0.95 0.25 asserted 1.00
Finally, based on our experimental findings, we identify several insights, providing directions for the continual self-evolution of agent harnesses. interpretation 0.67 0.89 0.14 asserted 0.96
Code is available at https://github.com/THU-Agent/continual-harness-evolution. contribution 0.89 0.43 0.02 asserted 0.88

What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 1.00

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-04: 1 request, 8,691 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (5)
Figure 1 of the paper
Figure 1. Overview of our study on continual harness self-evolution. opers use AI agents for writing, editing, or generating code, 62.2% for debugging or troubleshooting, and 50.6% for writing or maintaining tests Stack Exchange Inc. (2026). In software development, a user may first ask a coding agent to add a new feature and later ask it to write test files. Inspired by this observation, we propose continual harness self-evolution, which refers to successively evolving a harness across different task types, in contrast to evolving a harness independently for a single task. However, to the…
Figure 2 of the paper
Figure 2. Harness evolution on single dataset. Performance of AHE and Meta-Harness over successive evolution rounds on SWE-Bench, SWT-Bench, and FeatureBench. The annotations report the best-performing harness identified during each evolution process. 0 25 50 75 100 Performance(%) AHE Initial Iter 3 Iter 5 SWE → SWT SWE → Feature SWT → SWE SWT → Feature Feature → SWT Feature → SWE 0 25 50 75 100 Performance(%) Meta-Harness Initial Iter 3 Iter 5
Figure 3 of the paper
Figure 3. Catastrophic forgetting during continual harness self-evolution across six directions. For each direction, the harness evolved on a single dataset is continuously evolved on another dataset, with its earlier-dataset performance evaluated initially and after 3 and 5 additional continual evolution iterations (Initial, Iter 3, and Iter 5). Labels on evolved bars show the performance score and relative performance drop (𝜌). 4.1 RQ1: Catastrophic Forgetting During Continual Harness Self-Evolution To investigate whether continual harness self-evolution leads to catastrophic forgetting, we…
Figure 4 of the paper
Figure 4. Examples of Harness Conflicts in Continual Harness Self-Evolution. mance from the initial score of 82.0% to 100.0% on the randomly sampled 50 tasks. On FeatureBench, both methods eventually achieve an improvement of more than 10 percentage points. We circle the highest-scoring point in each line plot in
Figure 5 of the paper
Figure 5. Case study of the harness modification introduced during evolution from SWE-Bench to SWT-Bench. 14

Figures are the authors', reproduced from the paper.

References (51)
  1. [1] Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl - Ong, Arnav Singhvi, Herumb Shandilya, Michael J. Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alex Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. GEPA: reflective pr... on Pith
  2. [6] Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. Mem0: Building production-ready AI agents with scalable long-term memory. In In \^ e s Lynce, Nello Murano, Mauro Vallati, Serena Villata, Federico Chesani, Michela Milano, Andrea Omicini, and Mehdi ... on Pith
  3. [9] Che Jiang, Jincheng Zhong, Yu Fu, Kai Tian, Junlin Yang, Kaikai Zhao, Yuchong Wang, Tianwei Luo, Weizhi Wang, Yuxin Zuo, Guoli Jia, Xingtai Lv, Dianqiao Lei, Sihang Zeng, Yuru Wang, Zhenzhao Yuan, Xinwei Long, Ermo Hua, Can Ren, Xin Jiang, Shulei Xie, Yuanchun Zheng, Youbang S... on Pith
  4. [10] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7... on Pith
  5. [17] Niels M \" u ndler, Mark Niklas M \" u ller, Jingxuan He, and Martin T. Vechev. Swt-bench: Testing and validating real-world bug-fixes with code agents. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), A... on Pith
  6. [21] Stack Exchange Inc. Tasks delegated to ai agents. Stack Overflow Developer Survey 2026, 2026. URL https://survey.stackoverflow.co/2026/knowledge/data/know-agent-delegate on Pith
  7. [22] Wei Tao, Yucheng Zhou, Yanlin Wang, Wenqiang Zhang, Hongyu Zhang, and Yu Cheng. MAGIS: llm-based multi-agent framework for github issue resolution. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), Advanc... on Pith
  8. [23] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Trans. Mach. Learn. Res., 2024, 2024. URL https://openreview.net/forum?id=ehfRiF0R3a on Pith
  9. [24] Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, and et al. Op... on Pith
  10. [26] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. ... on Pith
  11. [30] Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Kamanuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, Urmish Thakker, James Y. Zou, and Kunle Olukotun. Agentic context engineering: Evolving contexts for self-improving language models. In Car... on Pith
  12. [34] Qixing Zhou, Jiacheng Zhang, Haiyang Wang, Rui Hao, Jiahe Wang, Minghao Han, Yuxue Yang, Shuzhe Wu, Feiyang Pan, Lue Fan, Dandan Tu, and Zhaoxiang Zhang. Featurebench: Benchmarking agentic coding for complex feature development. In Carl Vondrick, Bharath Hariharan, Colin Raffe... on Pith
  13. [35] Deep Learning , author =. 2016 , publisher = on Pith
  14. [36] Attention Is All You Need , author =. Advances in Neural Information Processing Systems , year = on Pith
  15. [37] Hinton, Geoffrey E. and Osindero, Simon and Teh, Yee Whye , journal =. A Fast Learning Algorithm for Deep Belief Nets , volume = on Pith
  16. [38] Self-Improving Agents in the Era of Experience: A Survey of Self- to Meta-Evolution , author=. OpenReview Archive , year= on Pith
  17. [39] Jiahang Lin and Shichun Liu and Chengjun Pan and Lizhi Lin and Shihan Dou and Xuanjing Huang and Hang Yan and Zhenhua Han and Tao Gui , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2604.25850 , eprinttype =. 2604.25850 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2604.25850
  18. [40] Yoonho Lee and Roshen Nair and Qizheng Zhang and Kangwook Lee and Omar Khattab and Chelsea Finn , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2603.28052 , eprinttype =. 2603.28052 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2603.28052
  19. [41] Hangfan Zhang and Shao Zhang and Kangcong Li and Chen Zhang and Yang Chen and Yiqun Zhang and Lei Bai and Shuyue Hu , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2606.09498 , eprinttype =. 2606.09498 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2606.09498
  20. [42] Tingyang Chen and Shuo Lu and Kang Zhao and Weicheng Meng and Hanlin Teng and Tianhao Li and Chao Li and Xule Liu and Jian Liang and Zhizhong Zhang and Yuan Xie and Heng Qu and Kun Shao and Jian Luan , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2606.14249 , epr... on PithPith's reviewarXivdoi:10.48550/arxiv.2606.14249
show 31 more
  1. [43] Qizheng Zhang and Changran Hu and Shubhangi Upasani and Boyuan Ma and Fenglu Hong and Vamsidhar Kamanuru and Jay Rainton and Chen Wu and Mengmeng Ji and Hanchen Li and Urmish Thakker and James Y. Zou and Kunle Olukotun , editor =. Agentic Context Engineering: Evolving Contexts... on Pith
  2. [44] Lakshya A. Agrawal and Shangyin Tan and Dilara Soylu and Noah Ziems and Rishi Khare and Krista Opsahl. The Fourteenth International Conference on Learning Representations,. 2026 , url = on Pith
  3. [45] Hanrong Zhang and Shicheng Fan and Henry Peng Zou and Yankai Chen and Zhenting Wang and Jiayu Zhou and Chengze Li and Wei. CoEvoSkills: Self-Evolving Agent Skills via Co-Evolutionary Verification , journal =. 2026 , url =. doi:10.48550/ARXIV.2604.01687 , eprinttype =. 2604.016... on PithPith's reviewarXivdoi:10.48550/arxiv.2604.01687
  4. [46] Peng Xia and Jianwen Chen and Hanyang Wang and Jiaqi Liu and Kaide Zeng and Yu Wang and Siwei Han and Yiyang Zhou and Xujiang Zhao and Haifeng Chen and Zeyu Zheng and Cihang Xie and Huaxiu Yao , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2602.08234 , eprinttype... on PithPith's reviewarXivdoi:10.48550/arxiv.2602.08234
  5. [47] Huichi Zhou and Yihang Chen and Siyuan Guo and Xue Yan and Kin Hei Lee and Zihan Wang and Ka Yiu Lee and Guchun Zhang and Kun Shao and Linyi Yang and Jun Wang , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2508.16153 , eprinttype =. 2508.16153 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2508.16153
  6. [48] Prateek Chhikara and Dev Khant and Saket Aryan and Taranjeet Singh and Deshraj Yadav , editor =. Mem0: Building Production-Ready. 2025 , url =. doi:10.3233/FAIA251160 , timestamp = on Pithdoi:10.3233/faia251160
  7. [49] Qinwen Yang and Liyuan Wang and Joerg Wicker and Gillian Dobbie , title =. Neural Networks , volume =. 2026 , url =. doi:10.1016/J.NEUNET.2025.108226 , timestamp = on Pithdoi:10.1016/j.neunet.2025.108226
  8. [50] Gail A. Carpenter and Stephen Grossberg , title =. Comput. Vis. Graph. Image Process. , volume =. 1987 , url =. doi:10.1016/S0734-189X(87)80014-2 , timestamp = on Pithdoi:10.1016/s0734-189x(87)80014-2
  9. [51] Carlos E. Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik R. Narasimhan , title =. The Twelfth International Conference on Learning Representations,. 2024 , url = on Pith
  10. [52] Niels M. SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents , booktitle =. 2024 , url = on Pith
  11. [53] Qixing Zhou and Jiacheng Zhang and Haiyang Wang and Rui Hao and Jiahe Wang and Minghao Han and Yuxue Yang and Shuzhe Wu and Feiyang Pan and Lue Fan and Dandan Tu and Zhaoxiang Zhang , editor =. FeatureBench: Benchmarking Agentic Coding for Complex Feature Development , booktit... on Pith
  12. [54] James Kirkpatrick and Razvan Pascanu and Neil C. Rabinowitz and Joel Veness and Guillaume Desjardins and Andrei A. Rusu and Kieran Milan and John Quan and Tiago Ramalho and Agnieszka Grabska. Overcoming catastrophic forgetting in neural networks , journal =. 2016 , url =. doi:... on PithPith's reviewarXivdoi:10.48550/arxiv.1612.00796
  13. [55] Junhao Zheng and Chengming Shi and Xidi Cai and Qiuke Li and Duzhen Zhang and Chenxing Li and Dong Yu and Qianli Ma , title =. 2026 , url =. doi:10.1109/TPAMI.2025.3650546 , timestamp = on Pithdoi:10.1109/tpami.2025.3650546
  14. [56] Guanzhi Wang and Yuqi Xie and Yunfan Jiang and Ajay Mandlekar and Chaowei Xiao and Yuke Zhu and Linxi Fan and Anima Anandkumar , title =. Trans. Mach. Learn. Res. , volume =. 2024 , url = on Pith
  15. [57] Yitao Liu and Chenglei Si and Karthik R. Narasimhan and Shunyu Yao , editor =. Contextual Experience Replay for Self-Improvement of Language Agents , booktitle =. 2025 , url =. doi:10.18653/V1/2025.ACL-LONG.694 , timestamp = on Pithdoi:10.18653/v1/2025.acl-long.694
  16. [58] Yuli Qiu and Yutong Li and Wei Su and Zeming Liu and Wanxiang Che and Heyan Huang and Haifeng Wang and Yuang Guo , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2609.12655 , eprinttype =. 2609.12655 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2609.12655
  17. [59] Junhao Zheng and Xidi Cai and Qiuke Li and Duzhen Zhang and Zhong. LifelongAgentBench: Evaluating. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2505.11942 , eprinttype =. 2505.11942 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2505.11942
  18. [60] Xingze Gao and Chuanrui Hu and Hongda Chen and Pengfei Yao and Zhao Wang and Yi Bai and Zhengwei Wu and Yunyun Han and Xiaofeng Cong and Jie Gui and Yafeng Deng and Teng Li , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2607.05202 , eprinttype =. 2607.05202 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2607.05202
  19. [61] Yiheng Shu and Bernal Jim. AgentCL: Toward Rigorous Evaluation of Continual Learning in Language Agents , journal =. 2026 , url =. doi:10.48550/ARXIV.2606.02461 , eprinttype =. 2606.02461 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2606.02461
  20. [62] Parth Asawa and Christopher M. Glaze and Gabriel Orlanski and Ramya Ramakrishnan and Benji Xu and Asim Biswal and Vincent Sunn Chen and Frederic Sala and Matei Zaharia and Joseph E. Gonzalez , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2606.05661 , eprinttype =... on PithPith's reviewarXivdoi:10.48550/arxiv.2606.05661
  21. [63] Thomas Joshi and Shayan Chowdhury and Fatih Uysal , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2507.00014 , eprinttype =. 2507.00014 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2507.00014
  22. [64] Dong Yan and Jian Liang and Dapeng Hu and Ran He and Nicholas Jing Yuan and Qi Zhang and Tieniu Tan , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2608.00155 , eprinttype =. 2608.00155 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2608.00155
  23. [65] Yuhao Zhan and Bingxiang He and Zecong Tang and Chaojun Xiao , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2608.14441 , eprinttype =. 2608.14441 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2608.14441
  24. [66] John Yang and Carlos E. Jimenez and Alexander Wettig and Kilian Lieret and Shunyu Yao and Karthik Narasimhan and Ofir Press , editor =. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering , booktitle =. 2024 , url = on Pith
  25. [67] Junwei Liu and Kaixin Wang and Yixuan Chen and Xin Peng and Zhenpeng Chen and Lingming Zhang and Yiling Lou , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2409.02977 , eprinttype =. 2409.02977 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2409.02977
  26. [68] Xingyao Wang and Boxuan Li and Yufan Song and Frank F. Xu and Xiangru Tang and Mingchen Zhuge and Jiayi Pan and Yueqi Song and Bowen Li and Jaskirat Singh and Hoang H. Tran and Fuqiang Li and Ren Ma and Mingzhang Zheng and Bill Qian and Yanjun Shao and Niklas Muennighoff and Y... on Pith
  27. [69] Wei Tao and Yucheng Zhou and Yanlin Wang and Wenqiang Zhang and Hongyu Zhang and Yu Cheng , editor =. Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2... on Pith
  28. [70] Islem Bouzenia and Premkumar T. Devanbu and Michael Pradel , title =. 47th. 2025 , url =. doi:10.1109/ICSE55347.2025.00157 , timestamp = on Pithdoi:10.1109/icse55347.2025.00157
  29. [71] Xuying Ning and Katherine Tieu and Dongqi Fu and Tianxin Wei and Zihao Li and Yuanchen Bei and Jiaru Zou and Mengting Ai and Zhining Liu and Ting. Code as Agent Harness , journal =. 2026 , url =. doi:10.48550/ARXIV.2605.18747 , eprinttype =. 2605.18747 , timestamp = on PithPith's reviewarXivdoi:10.48550/arxiv.2605.18747
  30. [72] Tasks delegated to AI agents , year= on Pith
  31. [73] DeepSeek. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence , journal =. 2026 , url =. doi:10.48550/ARXIV.2606.19348 , eprinttype =. 2606.19348 , timestamp = on Pithdoi:10.48550/arxiv.2606.19348
Cited by 1 Pith paper

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment