Pith. sign in

REVIEW 5 major objections 5 minor 13 references

From Knowledge to Noise: CTIM-Rover and the Pitfalls of Episodic Memory in Software Engineering Agents

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding cross-task memory to a software-repair agent did not help and often hurt it.

desk verdict A useful negative result undermined by a missing control: CTIM-Rover adds Reflexion to AutoCodeRover, so the reported 'memory hurts' conclusion is not actually isolated. read the letter →

arxiv 2505.23422 v1 pith:JYGKGUR2 submitted 2025-05-29 cs.SE cs.AI

Tobias Lindenbauer , Georg Groh , Hinrich Schütze This is my paper
classification cs.SEcs.AI
keywords softwareengineeringagentsepisodicmemorycross-task-instanceexperientiallearningAutoCodeRoverSWE-benchVerifiedin-contextknowledgedistillation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CTIM-Rover is an attempt to give a state-of-the-art bug-repair agent, AutoCodeRover, a cross-task-instance memory: a compact set of distilled lessons from 236 previously solved GitHub issues, plus, in some configurations, a similar past trajectory. The paper's claim is that this memory does not transfer to real software engineering: on a 45-issue stratified subset of SWE-bench Verified, every CTIM-Rover configuration matched or trailed the plain baseline, and the memory-only configuration dropped resolution from 42% to 31%. The authors take the result as evidence that experiential-learning frameworks that succeed on simpler benchmarks do not scale to long-horizon repository repair, and they trace the failure to noisy, unfiltered memory items that bias the agent's first steps. A sympathetic reader would care because reusing experience across tasks is an attractive, low-cost way to improve agents; if the paper is right, naive injection of distilled memory can be actively harmful until relevance filtering is solved.

What carries the argument

The load-bearing object is the Cross-Task-Instance Memory (CTIM), an episodic memory of distilled lessons from previously solved bug-fixing tasks. It is built in two knowledge-distillation phases: first, a reasoning-focused language model extracts general software-engineering rules from pairs and success-failure tuples of 236 successful trajectories; second, it extracts repository-specific rules (structure, conventions, failure modes) conditioned on the general rules, capped at $\lceil\sqrt{n}\rceil$ items. During evaluation the full CTIM is injected into the prompt at the start of a trajectory; the 'Exemplar only' variant instead retrieves the single most task-similar past trajectory from a Milvus index of Code-T5 problem-statement embeddings using a 90% cosine threshold. The mechanism is meant to let the agent skip exploration and reuse repository knowledge, but in practice the injected text becomes part of the model's context and can dominate or misdirect its early decisions.

What would settle it

Run all configurations on the full 500-instance SWE-bench Verified with several independent runs and report confidence intervals: if CTIM-Rover's success rate is statistically indistinguishable from or better than AutoCodeRover's, the paper's central claim is falsified. A complementary test would replace the static full-CTIM prompt with per-turn embedding retrieval of relevant items; if that beats AutoCodeRover, the harm comes from presentation or relevance rather than from cross-task memory per se.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a negative result about episodic memory in software engineering agents. AutoCodeRover resolves 42% of the 45 test issues; CTIM-Rover with general plus repository-level memory resolves 40%; with only the general/repository memory it resolves 31%; with only an exemplar trajectory, 40%. The repository-level memory did not help even on django, the repository the training trajectories were most skewed toward, and a qualitative trace shows a single memory item containing the word 'clean' diverting the agent from the function that actually needed the patch; removing that item let the system solve the issue. Adding an arbitrary capitalized item ('GRANDMA LIKES PASTA') to the memory restored success on another sample, which the paper reads as evidence that memory acts as surface-level noise that biases early exploration rather than as reusable knowledge. The paper concludes that the cross-task-instance memory approach, as implemented, does not generalize to software engineering and that the failures point toward retrieval of relevant memory items at each turn rather than a full static memory dump.

Load-bearing premise

The central comparison rests on a 45-instance test set in which the baseline solved 19 issues and the best memory configuration solved 18; if that gap is sampling noise, the claim that the memory degrades performance collapses.

Editorial extensions

If this is right

  • Unfiltered episodic memory can make an agent worse than no memory at all: the memory-only configuration resolved 31% of issues versus 42% for the unmodified baseline.
  • The benefit of experiential-learning methods on simpler benchmarks (web shopping, household tasks, multi-hop QA) does not by itself transfer to long-horizon repository debugging.
  • Coverage is not the bottleneck: even django, the most heavily represented repository in the training trajectories, did not benefit from repository-level memory, so item relevance and presentation matter more than how much memory is available.
  • Retrieving and injecting a task-similar exemplar trajectory (40%) also failed to beat the baseline, so merely adding a similar example is not sufficient.
  • A likely path forward is per-turn retrieval of a small set of relevant memory items rather than one static injection, the direction the paper points to and leaves untested.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper reports percentages without error bars; 45 test instances make 42% versus 40% a 19-versus-18 resolved-instance gap, so the strongest defensible claim from the table is 'no measured improvement,' not 'proven degradation.'
  • The paper's noise diagnosis rests on two hand-picked cases plus the 'GRANDMA LIKES PASTA' probe; a larger study that deliberately injects unrelated memory items and measures the shift in first-action distribution would test whether surface format, not semantic content, causes the bias.
  • The paper explicitly does not run the retrieval-based variant it proposes, so the recommendation that per-turn embedding retrieval would fix the degradation is a conjecture, not a demonstrated result.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper introduces CTIM-Rover, an AutoCodeRover-based agent augmented with a Cross-Task-Instance Memory (CTIM) distilled from successful trajectories, including a repository-level variant. The authors evaluate several configurations (full CTIM, general CTIM only, repo-level CTIM only, exemplar only) on a 45-instance subset of SWE-bench Verified and report that none outperforms vanilla AutoCodeRover, with the full configuration scoring 40% versus a 42% baseline. Based on two qualitative examples, they attribute the degradation to noisy CTIM items and conclude that ExpeL and DoT-Bank do not scale to software engineering.

Significance. If the central result were established, this would be a useful negative result: it would caution against unconditional transfer of episodic-memory and experiential-learning methods from toy environments to SWE-bench, and the released code and detailed prompts would support follow-up work on retrieval-based memory. The paper also has methodological strengths: a held-out test split stratified on annotated difficulty and false-negative features, careful documentation of trajectory collection, and an explicit limitations section. However, the main attribution is currently confounded and statistically underpowered, so the significance is conditional on revision.

major comments (5)
  1. [§4, Figure 1, §3.1] The comparison is confounded: CTIM-Rover differs from AutoCodeRover not only by the memory but also by Reflexion full-trajectory retries and a reduced patch-generation retry budget. Section 3.1 states that during training the patch-generation retries are reduced from three to two, and Figure 1 shows that CTIM-Rover adds Reflexion-style self-reflection with up to three retries. The paper never reports whether the evaluation runs use the same Reflexion/retry configuration, and it provides no AutoCodeRover+Reflexion control. Consequently, every row of Table 1 attributes the full difference to CTIM, but the observed drops (e.g., 42% to 31%) could be caused by the retry loop, the retry budget, or the longer contexts (Table 3), not by the distilled knowledge. This blocks the headline 'memory degrades performance' claim.
  2. [§5, Table 1] No uncertainty quantification is provided for the success rates. With n=45, the 42% versus 40% difference is one resolved instance (19 versus 18) and is within sampling noise; no confidence intervals, error bars, or significance tests are reported anywhere. The cross-configuration comparisons (31% to 40%) and the 'degradation' language are therefore not supported by the data as reported. At minimum, the authors should report exact counts, intervals, and a paired test such as McNemar's test, since all configurations are evaluated on the same 45 instances.
  3. [Appendix D.3, Figure 24] The 'All psf Repository-Level CTIM Items' list contradicts the claimed repository-specific construction in Section 4.1. Three of the four psf items refer to Django-specific components ('RegisterLookupMixin', 'returning_fields', and 'admin logic in Django'), while only the first item concerns requests/models.py. This indicates that the repository-level knowledge base was contaminated with items from other repositories, either through the distillation pipeline or the prompt bookkeeping. This is a concrete flaw in the very component the paper identifies as harmful, and it weakens the claim that the repo-level CTIM was properly constructed; the authors should explain or fix this before the repository-level attribution can be evaluated.
  4. [§5.1] The causal diagnosis rests on two qualitative examples in which the authors manually remove one item or add an arbitrary item and observe behavior changes. There is no systematic ablation (e.g., removing all items containing certain tokens, permuting item order, or repeating runs to control for LLM stochasticity), and the two examples are not sufficient to support the general conclusion that noise in CTIM items is 'the likely source' of the degradation. The evidence is suggestive for a future retrieval-based design but not load-bearing for the paper's main claim.
  5. [Abstract and §6] The conclusion that 'neither ExpeL nor DoT-Bank scale to real-world SE problems' overstates what was tested. The paper evaluates CTIM-Rover, an ExpeL-inspired method with substantial modifications (repository-level KD, MoE-style grouping, Reflexion retries, size caps), and does not run ExpeL or DoT-Bank themselves on SWE-bench. The negative result can only support a claim about this instantiation, not about the original methods.
minor comments (5)
  1. [Figure 2 caption] The caption reads 'CITM-Rover' and should be 'CTIM-Rover'.
  2. [Figure 4 and Appendix D.2] Figure 4 contains 'T est-Split', and several prompt templates in Appendix D.2 contain typos such as 'didn 't in introduce' (Figures 12 and 14).
  3. [§4.1 and §4.2] The size cap c(n)=⌈√n⌉ and the 90% cosine threshold are free parameters with no sensitivity analysis; since the result is a null/negative effect, readers cannot tell whether another reasonable choice would change the outcome.
  4. [§4] The term 'Mixture-Of-Experts inspired' is not operationalized; the paper describes a two-phase knowledge distillation process with repository grouping, not an MoE model, and the analogy is unclear.
  5. [Table 3] The CTIM-Rover configurations consume substantially more tokens than AutoCodeRover, so a cost-adjusted comparison (e.g., success per dollar or per token) would strengthen the negative result and help readers judge the practical trade-off.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the evaluation uses a held-out test set and a memory built from training-trajectory experiences, with no parameter fitted to the test predictions.

full rationale

The paper's derivation chain is an empirical comparison: AutoCodeRover is augmented with a Cross-Task-Instance Memory distilled from successful trajectories on a training partition of SWE-bench Verified, and then evaluated on a disjoint 45-instance test partition (Appendix A). The memory is not defined in terms of the test outcomes, no parameter is fitted to the test set, and the central comparison rows in Table 1 are all held-out evaluations. The conclusion that CTIM degrades performance is a reported experimental observation, not a quantity that is equal to its inputs by construction. The skeptical concerns about the Reflexion retry configuration, the reduced patch-generation retries during training, the small test-set size, and the lack of a Reflexion-only control are threats to causal attribution and statistical validity, but they are not circularity: they do not make the output equivalent to the input by definition. The only potential leakage-related issue would be if the 53 supplementary SWE-bench Lite trajectories overlapped with the 45 test instances, but the paper does not establish such overlap, and asserting it would require external information rather than an exhibited reduction from the paper's own text. Self-citation is not load-bearing here: the paper builds on AutoCodeRover and ExpeL as external prior systems and does not invoke any author-specific uniqueness theorem or fitted ansatz. Overall, the derivation is self-contained against an external benchmark and the reported negative result is a genuine experimental finding, so no circular step is identified.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The central claim rests on the quality of the 45-sample evaluation and on the distillation procedure. Several design choices (memory cap, retrieval threshold, importance doubling) are hand-set. The paper's causal analysis adds assumptions about the representativeness of two qualitative examples.

free parameters (3)
  • CTIM maximum size cap = c(n) = ceil(sqrt(n))
    Limits memory to a concise set; affects which rules survive distillation and therefore agent behavior.
  • Exemplar retrieval cosine threshold = 0.9
    Controls when a past trajectory is injected; with 0.9, roughly 62% of test samples receive an exemplar.
  • Initial importance of CTIM items = doubled relative to ExpeL
    Changed to account for longer gaps between applicable instances; influences which items get downvoted out of the memory.
assumptions (4)
  • domain assumption The stratified split on difficulty and false_negative annotations yields a representative test set for SWE-bench Verified.
    Used to build the train and test sets (Appendix A). If stratification fails, the reported results may not reflect the benchmark.
  • domain assumption The removed repositories (pylint, astropy, pydata/xarray) and ~10% of samples are not systematically different from those kept.
    Environment setup issues led to their removal (Section 3, Limitations). If these samples differ, the test set is biased.
  • domain assumption The o1 model can extract accurate, non-redundant, and transferable rules from successful trajectories.
    The CTIM is built by o1 distillation (Section 4.1). The paper's own analysis shows some distilled items are misleading, so this assumption is questionable.
  • ad hoc to paper The two qualitative examples in Section 5.1 are representative of the general failure mode.
    The noise conclusion is drawn from these two cases, with no broader quantitative support.
invented entities (2)
  • Cross-Task-Instance Memory (CTIM)
    purpose: Stores distilled rules from past successful trajectories to guide future bug-fixing tasks.
    The paper provides no external validation of the memory's correctness; on the contrary, one CTIM item (containing 'clean') is shown to mislead the agent (Section 5.1).
  • Repository-level CTIM
    purpose: A repository-specific subset of CTIM with rules about a particular codebase's structure and patterns.
    Inherits the same lack of external validation; its items are found in the qualitative analysis to bias the agent (e.g., django item in Figure 3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Knowledge to Noise: CTIM-Rover and the Pitfalls of Episodic Memory in Software Engineering Agents." pith.science (2026). https://pith.science/paper/JYGKGUR2

@misc{pith2026250523422,
  author       = {Pith},
  title        = {Pith review of: From Knowledge to Noise: CTIM-Rover and the Pitfalls of Episodic Memory in Software Engineering Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JYGKGUR2}},
  note         = {Machine review of arXiv:2505.23422}
}
read the original abstract

We introduce CTIM-Rover, an AI agent for Software Engineering (SE) built on top of AutoCodeRover (Zhang et al., 2024) that extends agentic reasoning frameworks with an episodic memory, more specifically, a general and repository-level Cross-Task-Instance Memory (CTIM). While existing open-source SE agents mostly rely on ReAct (Yao et al., 2023b), Reflexion (Shinn et al., 2023), or Code-Act (Wang et al., 2024), all of these reasoning and planning frameworks inefficiently discard their long-term memory after a single task instance. As repository-level understanding is pivotal for identifying all locations requiring a patch for fixing a bug, we hypothesize that SE is particularly well positioned to benefit from CTIM. For this, we build on the Experiential Learning (EL) approach ExpeL (Zhao et al., 2024), proposing a Mixture-Of-Experts (MoEs) inspired approach to create both a general-purpose and repository-level CTIM. We find that CTIM-Rover does not outperform AutoCodeRover in any configuration and thus conclude that neither ExpeL nor DoT-Bank (Lingam et al., 2024) scale to real-world SE problems. Our analysis indicates noise introduced by distracting CTIM items or exemplar trajectories as the likely source of the performance degradation.

Figures

Figures reproduced from arXiv: 2505.23422 by the authors.

Figure 1
Figure 1. CTIM-Rover Overview. Figure inspired by ExpeL (Zhao et al., 2024). CTIM-Rover first gathers new experiences on the train set of SWE-bench Verified which we introduce in Section 3 (details in Appendix A). Then, it combines these experiences with existing experiences of AutoCodeRover (Zhang et al., 2024) on SWE-bench Lite (Jimenez et al., 2023). Next, it distills high-level and repository-level knowledge from these ex… view at source ↗
Figure 2
Figure 2. CITM-Rover Knowledge Distillation (KD). Key departure from ExpeL (Zhao et al., 2024) in blue. Top: (1) Distill generally applicable SE knowledge from pairs of successful trajectories from different task instances and (2) tuples of a successful task instance and its self-reflection retries. Bottom: (3) Use the generally applicable knowledge and past experience to distill repository-level knowledge from pairs of succe… view at source ↗
Figure 3
Figure 3. Excerpt of the repository-level CTIM item that biased our system toward investigating the incorrect clean function, demonstrating how seemingly innocu￾ous knowledge can misguide the agent. ration. The problem statement of this sample ex￾plicitly mentions the constant FIXTURE_DIRS and AutoCodeRover correctly searches the repository for this constant. However, CTIM-Rover with the “CTIM only” configuration does not. We… view at source ↗
Figures from the paper (21 more)
Figure 4
Figure 4. Figure 4: The distribution of repositories across our [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The distribution of repositories across success [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Key differences in our prompting strategies [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: The system prompt we use for both KD phases and all KD settings. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: The CTIM capacity warning prompt we use in both KD phases and all KD settings. [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: The CTIM capacity information prompt we use in both KD phases and all KD settings. [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: The CTIM operations prompt we use in the high level KD phase and both its KD settings. [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 11
Figure 11. Figure 11: The CTIM operations prompt we use in the high level KD phase and both its KD settings continued. [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: The prompt describing the success-failure trajectory pair KD setting for the high level KD phase. [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]
Figure 13
Figure 13. Figure 13: The prompt describing the success-failure trajectory pair [PITH_FULL_IMAGE:figures/full_fig_p012_13.png]
Figure 14
Figure 14. Figure 14: The prompt describing the sets of successful trajectories KD setting for the high level KD phase. [PITH_FULL_IMAGE:figures/full_fig_p012_14.png]
Figure 15
Figure 15. Figure 15: The prompt describing the sets of successful trajectories [PITH_FULL_IMAGE:figures/full_fig_p013_15.png]
Figure 16
Figure 16. Figure 16: The CTIM operations prompt we use in the repository-level KD phase and both its KD settings. [PITH_FULL_IMAGE:figures/full_fig_p013_16.png]
Figure 17
Figure 17. Figure 17: The CTIM operations prompt we use in the repository-level KD phase and both its KD settings. Continued. Success-Failure Trajectory KD Setting - Repository-Level KD (Phase 2) You are given a set of successful task trajectories that relate to fixing issues the real-worl…
Figure 18
Figure 18. Figure 18: The prompt describing the sets of successful trajectories KD setting for the repository-level KD phase. [PITH_FULL_IMAGE:figures/full_fig_p014_18.png]
Figure 19
Figure 19. Figure 19: The prompt describing the sets of successful trajectories [PITH_FULL_IMAGE:figures/full_fig_p015_19.png]
Figure 20
Figure 20. Figure 20: The prompt describing the success-failure trajectory pair [PITH_FULL_IMAGE:figures/full_fig_p015_20.png]
Figure 21
Figure 21. Figure 21: The prompt describing the success-failure trajectory pair [PITH_FULL_IMAGE:figures/full_fig_p016_21.png]
Figure 22
Figure 22. Figure 22: Four random general CTIM samples. Sample Django Repository-Level CTIM Items - When refactoring special-case or zero-quantity paths in the app\u2019s code (like max_post_process_passes=0), skip irrelevant steps entirely to avoid referencing uninitialized variables. If …
Figure 23
Figure 23. Figure 23: Four random django CTIM samples [PITH_FULL_IMAGE:figures/full_fig_p016_23.png]
Figure 24
Figure 24. Figure 24: All psf repository-level CTIM samples [PITH_FULL_IMAGE:figures/full_fig_p017_24.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

13 extracted references · 11 canonical work pages

  1. [1]

    Reasoning and planning strategies that serve as guiding signals for future task attempts, especially with respect to identifying the locations of buggy code effectively

    work page

  2. [3]

    Figure 12: The prompt describing the success-failure trajectory pair KD setting for the high level KD phase

    Common pitfalls and error patterns in software engineering that are relevant to identifying and fixing buggy code. Figure 12: The prompt describing the success-failure trajectory pair KD setting for the high level KD phase. Success-Failure Trajectory KD Setting - General KD (Phase 1) continued Key requirements for rules: - DO NOT suggest testing the imple...

    work page

  3. [4]

    The application domain of the project (e.g., Does the software handle images or text and what kind? Is it a command line application or does it have a GUI? Does it handle HTTP requests? Is it a highly technical, mathematical application?)

    work page

  4. [5]

    What are common errors or potential pitfalls in these application domains?)

    Common edge cases or failure modes related to the project 's specific application domain. What are common errors or potential pitfalls in these application domains?). Key requirements for rules: - DO NOT suggest testing the implementation. The agent must generate correct patches on the first attempt by leveraging general and repository-specific rules iden...

    work page

  5. [7]

    Reasoning and planning strategies that serve as guiding signals for future task attempts, especially with respect to entifying the locations of buggy code effectively

    work page

  6. [8]

    Coding practices, patterns, and idioms that are generally applicable to writing high-quality, staff senior level code, to fix bugs

    work page

  7. [9]

    operation_type

    Common pitfalls and error patterns in software engineering that are relevant to identifying and fixing buggy code. Key requirements for rules: - DO NOT suggest testing the implementation. The agent using your ruleset is UNABLE to test its implementation. It must generate a correct patch on the first attempt. - DO NOT suggest reflecting on a past trajector...

    work page

  8. [10]

    Repository-level bug localization and environment exploration patterns that help locate relevant code sections quickly, including key file locations, module relationships

    work page

Show all 13 references
  1. [11]

    Repository-level coding conventions, architectural principles, design patterns, and implementation approaches that are consistently used across the codebase and should be followed when making changes

  2. [12]

    Repository-level error or exception handling strategies, including custom errors or exceptions

  3. [2017]

    Proceedings of the National Academy of Sciences, 114(13):3521–3526

    Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521–3526. Publisher: Proceed- ings of the National Academy of Sciences. Vijay Lingam, Behrooz Omidvar Tehrani, Sujay Sang- havi, Gaurav Gupta, Sayan Ghosh, Linbo Li...

    2024 arXiv

  4. [2024]

    psf” or “pallets

    Table 2 presents the p-values for the inves- tigated features. We find that the difficulty and false_negative features are statistically signifi- cantly associated with the resolution of an instance at a significance level ofα = 0.01. These fields cor- respond to the subjectiv...

    2025

  5. [2025]

    arXiv preprint

    SWE-smith: Scaling Data for Software Engi- neering Agents. arXiv preprint. ArXiv:2504.21798 [cs]. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Ben- gio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi...

    2018 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.