REVIEW 4 major objections 6 minor 34 references
This paper claims that a sequence of individually benign tasks and experiences can jointly erode a self-evolving LLM agent's safety boundary, and demonstrates it with EvoBreak at an average attack success rate of 86.12%.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-04 21:30 UTC pith:LIXJWEBK
load-bearing objection Solid, novel attack on self-evolving agents that deserves a serious referee, but the judge-based evaluation makes the 'benign in isolation' claim softer than advertised. the 4 major comments →
Benign Alone, Harmful Together: Exploiting Experience Composition in Self-Evolving LLM Agents
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central claim is that the 'benign alone, harmful together' effect is real and exploitable in self-evolving LLM agents. EvoBreak does not inject a malicious memory and never submits a malicious task; instead it repeatedly observes the experiences the victim distills, computes an experience residual that names which target-relevant requirements are still missing, submits another benign task to acquire that piece, and stops when coverage is sufficient. It then rewrites the original safety-sensitive query so that its structure aligns with the accumulated experiences, prompting the victim to retrieve and jointly apply them. The reported result is an average attack success rate of 86.1
What carries the argument
Two mechanisms carry the argument. The first is EvoBreak's closed-loop experience residual: an explicit textual planning state, recomputed after every interaction, that lists the target-relevant knowledge, procedures, or constraints still absent from the accumulated experiences; the next benign task is chosen to fill that residual, and the final query is generated conditioned on the full history. The second is BreakGym's latent scaffold: a structural template combining a decomposition principle, a composition topology, and a target domain, used only to synthesize training targets and to provide hints during training, so the policy learns to plan over requirement gaps rather than fixed task s
Load-bearing premise
The attack only works if the attacker can see the experience summaries the victim saves after each interaction; if that memory is hidden, the closed-loop replanning breaks and EvoBreak degrades to a fixed task sequence.
What would settle it
Run EvoBreak against a self-evolving agent whose distilled experiences are never exposed to the user. If attack success stays near the paper's fixed-decomposition baseline (roughly 60% in the w/o Replanning ablation) rather than the reported ~86%, the claim that experience-conditioned replanning drives the attack is falsified; if success stays high, the observability premise is not load-bearing.
If this is right
- Self-evolving agents that silently retain and reuse interaction-derived experiences inherit the attack surface even when every adversary-visible step is benign.
- Per-query and per-experience safety filters are insufficient; audits must evaluate the joint coverage of an accumulated experience set.
- Persistent-memory attacks no longer need direct write access or explicitly malicious memory records—only task submission plus observation of distilled experiences.
- The attack transfers across distinct evolution mechanisms, victim backbones, and pre-existing memory contents, making the underlying mechanism structural rather than model-specific.
Where Pith is reading between the lines
- A natural defense the paper leaves untested: hide or abstract the distilled experiences from the user. If EvoBreak cannot observe them, its replanning condition disappears and it should fall back to the fixed-sequence behavior the paper's own 'w/o Replanning' ablation shows at roughly 60% ASR.
- The same accumulation-and-activation pattern suggests ordinary, non-adversarial usage could gradually drift an agent's safety boundary, since many users will naturally cover the local requirements of sensitive tasks over time.
- The composition mechanism likely extends beyond safety to any persistent agent memory that can be jointly triggered by a rewritten request—privacy policies, tool-use rules, or preference profiles are candidate surfaces.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes EvoBreak, an attack against self-evolving LLM agents that accumulate persistent experiences from interaction trajectories. The central claim is that experiences that are individually benign can, when accumulated and jointly reused, weaken an agent's safety boundary. The adversary observes the experiences distilled from its own benign tasks, computes an 'experience residual,' replans the next benign task, and finally reformulates the target query to activate the accumulated experiences. Training uses BreakGym, a structure-first pipeline for generating decomposable safety-sensitive targets, followed by rejection-sampling SFT and Hint-guided GRPO. Experiments on ReasoningBank and SE-Agent, with GPT-5-mini and Llama-3.1-8B backbones, report an average ASR of 86.12% across JailbreakBench and HarmBench, outperforming five baselines, while maintaining high benignness as measured by an LLM judge.
Significance. If the central claim holds, this identifies a practically important attack surface: persistent experience memory, which is designed to improve agents, can be exploited without direct memory write access or explicitly malicious injected content. The closed-loop planning over observed experiences and the BreakGym structure-first synthesis pipeline are useful conceptual contributions, and the experimental matrix is broad. However, the empirical foundation rests entirely on an LLM judge used both for training rewards and for the final metrics, with no independent verification, no human audit, and no release of code, data, or judge outputs. The benignness constraint is defined in a way that makes 'benign in isolation' nearly tautological. As presented, the results are suggestive but not yet convincing evidence for the paper's main claim.
major comments (4)
- [§3.3, Eq. (8)-(11); §4.1] There is a circularity between the training objective and the evaluation metrics. The GRPO reward in Eq. (8) includes a benignness term B(t_i)+B(e^a_i) that is evaluated by the same type of LLM judge (GPT-5.4-mini) used to compute the final benignness metric in Eq. (11), and attack success S_atk in Eq. (8) is derived from the same score-threshold procedure as ASR in Eq. (10). The attack is therefore optimized to satisfy this particular judge, and the reported 86.12% ASR and high benignness may reflect judge-specific biases rather than genuine safety-relevant properties. The authors should re-evaluate all attack outputs with at least two independent judge families (e.g., a different proprietary model and an open-weight judge) and provide a human-annotated subset with agreement statistics. Without this, the headline quantitative claims are not independently verifiable.
- [§2.2, Eq. (2); Appendix C.2] The atomic benignness constraint is measured with a prompt that explicitly tells the judge to evaluate content 'without relying on any additional context or related interactions' and 'Do not infer malicious intent solely from its possible relationship to other interactions.' Under this protocol, any decomposed step can be phrased as a standalone benign request, so the paper's central claim that the attack operates through 'individually benign' experiences is close to tautological. What is needed is an external audit that evaluates whether a human, knowing the overall target, would still consider each component benign. Without such an audit, the paper does not actually demonstrate benign-in-isolation composition; it demonstrates that a context-free judge labels the components benign.
- [§2.3, Eq. (3)-(4); Table 2] The threat model assumes read-only observability of the victim's distilled experience e^a_i after every interaction. This assumption is load-bearing: EvoBreak's closed-loop residual planning in Eq. (4) conditions every next action on the observed experience text, and the 'w/o Replanning' ablation in Table 2 shows a large drop (e.g., 90.6% to 67.6% for Llama-3.1-8B, JailbreakBench, Math domain). Many deployed self-evolving agents may not expose internal distillation outputs to end users. The authors should state which deployments satisfy this gray-box assumption, and evaluate a weaker variant where the adversary observes only task success/failure or retrieved-memory snippets rather than the full extracted experience.
- [§4.1-§4.2] No code, data, or evaluation artifacts are released, and all results are aggregates over five runs without standard deviations, confidence intervals, or significance tests. Given that the entire evaluation depends on a proprietary LLM judge, the reported margins (e.g., +24.19 points over ReNeLLM) cannot be checked or re-judged by the community. The paper should release victim configurations, attack outputs, and judge transcripts, and at minimum report per-run variance and bootstrap confidence intervals for the main ASR and benignness numbers.
minor comments (6)
- [§3.1, Eq. (8)] In Eq. (8), the benignness term is divided by kζ, but it is not stated what happens if kζ=0. Also, the hyperparameters λ, β, ε, and M are introduced without sensitivity analysis; since these are free parameters, the robustness of the results to their values should be discussed.
- [§4.2, Table 1] The claim of an 'overall average ASR of 86.12%' should be shown explicitly with the averaging formula. Table 1 contains many numbers and no variance; adding per-cell standard deviations or a compact error-bar visualization would improve interpretability.
- [§3.2, Stage 3] The leakage-filtering step is described only as 'similarity to downstream evaluation prompts' with no threshold or measure. Since BreakGym is used to train an attack evaluated on JailbreakBench/HarmBench, details on the deduplication procedure are needed to rule out contamination.
- [Appendix B.5] The model names GPT-5.4-mini, Qwen3.5-9B, and Gemini 3 Flash are not standard public identifiers at the time of writing. Please provide exact model versions, access dates, and hosting details.
- [§2.3] The paper calls the threat model 'gray-box' and contrasts with 'query-only' attacks, but EvoBreak requires observing the victim's internal experiences. This is stronger than query-only access; the terminology should be clarified to avoid overstating stealth.
- [§4.3, Figure 3] The effectiveness-benignness Pareto plot would be more informative with error bars or at least per-run scatter points, especially because both axes are LLM-judge outputs.
Circularity Check
Benignness metric is the training reward; 'benign in isolation' is optimized, not independently demonstrated.
specific steps
-
fitted input called prediction
[Eq. (8) in Section 3.3 (Cascaded Optimization) vs. Eq. (11) in Appendix C.2 (Evaluation Prompt of Benignness); Section 3.3 rejection-sampling SFT filter]
"R(ζ) = S_atk(ζ) + λ/2kζ Σ [B(t_i)+B(e^a_i)] (Eq. 8); Ben. = 1/2K Σ [B(t_i)+B(e^a_i)]×100% (Eq. 11); 'We retain only rollouts in which all submitted tasks and induced experiences pass the benignness audit and the final target-stage attack succeeds.'"
The identical auditor B appears in the training reward (both the GRPO benignness term and the SFT filter) and in the evaluation metric for benignness. EvoBreak is explicitly optimized and filtered to maximize B, and the paper then reports the resulting B average as evidence that the attack 'maintains high benignness' and operates through 'individually benign attack-stage tasks.' The reported benignness is therefore the training objective itself, not an independent verification of the paper's central 'benign in isolation' premise. The ASR on held-out benchmarks retains independent content, which is why the circularity is partial rather than total.
full rationale
The main derivation chain is: train EvoBreak on BreakGym via rejection-sampling SFT and Hint-guided GRPO with reward Eq. (8); evaluate on JailbreakBench/HarmBench with ASR Eq. (10) and benignness Eq. (11). The central circular step is the benignness metric: the same B function is the benignness term in the training reward, the filter for SFT rollouts, and the evaluation metric. Thus the paper's evidence for its novel 'benign in isolation' claim reduces to the objective it optimized for, not an independent measurement. The ASR result is less circular because it is measured on held-out benchmarks and the victim models differ from the judge, but the reported ASR and benignness are not fully independent since the same judge family (GPT-5.4-mini) is used for both reward and evaluation. No load-bearing self-citation chain was found: the Yan et al. 2026 self-citation in the introduction is paired with Lai et al. 2026 and only motivates the attack surface, and no uniqueness theorem or ansatz is imported from the authors' prior work. The context-free benignness prompt makes 'benign in isolation' definitional, but the deeper problem is the train/eval overlap. Overall, the central attack-effectiveness claim has independent content, while the central novelty's benignness evidence is partly circular, giving a score of 6.
Axiom & Free-Parameter Ledger
free parameters (4)
- lambda (benignness weight in Eq. 8) =
not reported
- beta (KL penalty in Eq. 9) =
0.001
- epsilon (GRPO clip) =
0.2
- M (number of rollouts) =
8
axioms (4)
- domain assumption Victim exposes its distilled experiences to the adversary after each interaction.
- domain assumption Persistent experience memory is available at the target stage.
- domain assumption The LLM judge accurately measures both harmfulness and benignness.
- domain assumption Experience extraction in ReasoningBank and SE-Agent produces reusable experiences as assumed.
Cite this review
Pith. "Pith review of Benign Alone, Harmful Together: Exploiting Experience Composition in Self-Evolving LLM Agents." pith.science (2026). https://pith.science/paper/LIXJWEBK
@misc{pith2026260801759,
author = {Pith},
title = {Pith review of: Benign Alone, Harmful Together: Exploiting Experience Composition in Self-Evolving LLM Agents},
year = {2026},
howpublished = {\url{https://pith.science/paper/LIXJWEBK}},
note = {Machine review of arXiv:2608.01759}
}
read the original abstract
Self-evolving large language model agents improve their capabilities by distilling interaction trajectories into persistent experiences. Yet this mechanism introduces a new safety risk: experiences that are benign in isolation may jointly weaken an agent's safety boundary when accumulated and reused across sessions. Existing memory attacks typically require direct memory access or induce explicitly malicious records, limiting their stealthiness and applicability. We propose EvoBreak, an experience-conditioned sequential attack that operates through individually benign attack-stage tasks and induced experiences. EvoBreak repeatedly observes the experiences distilled by the victim, identifies uncovered target-relevant requirements, and adaptively acquires complementary experiences before reformulating the final query to activate them jointly. To support training, we introduce BreakGym, a structure-first synthesis pipeline that generates decomposable safety-sensitive targets with diverse dependency structures. EvoBreak is optimized using rejection-sampling supervised fine-tuning and Hint-guided GRPO. Experiments across self-evolving frameworks, victim backbones, pre-evolution domains, and safety benchmarks demonstrate that EvoBreak consistently outperforms existing attacks while maintaining high benignness. These results reveal benign experience composition as a persistent attack surface in self-evolving agents.
Figures
Reference graph
Works this paper leans on
-
[1]
Advances in Neural Information Processing Systems , volume=
Agentpoison: Red-teaming llm agents via poisoning memory or knowledge bases , author=. Advances in Neural Information Processing Systems , volume=
-
[2]
Advances in Neural Information Processing Systems , volume=
Memory injection attacks on LLM agents via query-only interaction , author=. Advances in Neural Information Processing Systems , volume=
-
[3]
OEP: Poisoning Self-Evolving LLM Agents via Locally Correct but Non-Transferable Experiences
OEP: Poisoning Self-Evolving LLM Agents via Locally Correct but Non-Transferable Experiences , author=. arXiv preprint arXiv:2605.18930 , year=
work page internal anchor Pith review Pith/arXiv arXiv
-
[4]
arXiv preprint arXiv:2512.16962 , year=
MemoryGraft: Persistent compromise of LLM agents via poisoned experience retrieval , author=. arXiv preprint arXiv:2512.16962 , year=
-
[5]
arXiv preprint arXiv:2602.15654 , year=
Zombie agents: Persistent control of self-evolving LLM agents via self-reinforcing injections , author=. arXiv preprint arXiv:2602.15654 , year=
-
[6]
Findings of the Association for Computational Linguistics: ACL 2026 , pages=
On Safety Risks in Experience-Driven Self-Evolving Agents , author=. Findings of the Association for Computational Linguistics: ACL 2026 , pages=
work page 2026
-
[7]
arXiv preprint arXiv:2502.14321 , year=
Beyond self-talk: A communication-centric survey of llm-based multi-agent systems , author=. arXiv preprint arXiv:2502.14321 , year=
-
[8]
arXiv preprint arXiv:2509.25140 , year=
Reasoningbank: Scaling agent self-evolving with reasoning memory , author=. arXiv preprint arXiv:2509.25140 , year=
-
[9]
arXiv preprint arXiv:2508.02085 , year=
Se-agent: Self-evolution trajectory optimization in multi-step reasoning with llm-based agents , author=. arXiv preprint arXiv:2508.02085 , year=
-
[10]
arXiv preprint arXiv:2601.07348 , year=
Controlled self-evolution for algorithmic code optimization , author=. arXiv preprint arXiv:2601.07348 , year=
-
[11]
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=
WebEvolver: Enhancing Web Agent Self-Improvement with Co-evolving World Model , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=
2025
-
[12]
arXiv preprint arXiv:2402.02716 , year=
Understanding the planning of llm agents: A survey , author=. arXiv preprint arXiv:2402.02716 , year=
-
[13]
Evo-Attacker: Memory-Augmented Reinforcement Learning for Long-Horizon Tool Attacks on LLM-MAS , author=. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
-
[14]
arXiv preprint arXiv:2402.04249 , year=
Harmbench: A standardized evaluation framework for automated red teaming and robust refusal , author=. arXiv preprint arXiv:2402.04249 , year=
-
[15]
Advances in Neural Information Processing Systems , volume=
A strongreject for empty jailbreaks , author=. Advances in Neural Information Processing Systems , volume=
-
[16]
International Conference on Learning Representations , volume=
Agentharm: A benchmark for measuring harmfulness of llm agents , author=. International Conference on Learning Representations , volume=
-
[17]
2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) , pages=
Jailbreaking black box large language models in twenty queries , author=. 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) , pages=. 2025 , organization=
2025
-
[18]
arXiv preprint arXiv:2410.02832 , year=
Flipattack: Jailbreak llms via flipping , author=. arXiv preprint arXiv:2410.02832 , year=
-
[19]
A wolf in sheep’s clothing: Generalized nested jailbreak prompts can fool large language models easily , author=. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=
work page 2024
-
[20]
Advances in Neural Information Processing Systems , volume=
Jailbreakbench: An open robustness benchmark for jailbreaking large language models , author=. Advances in Neural Information Processing Systems , volume=
- [21]
-
[22]
International Conference on Learning Representations , volume=
Livecodebench: Holistic and contamination free evaluation of large language models for code , author=. International Conference on Learning Representations , volume=
-
[23]
Advances in Neural Information Processing Systems , volume=
Mmlu-pro: A more robust and challenging multi-task language understanding benchmark , author=. Advances in Neural Information Processing Systems , volume=
-
[24]
arXiv preprint arXiv:2407.21783 , year=
The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=
-
[25]
2025 , howpublished =
2025
-
[26]
Qwen3.5: Towards Native Multimodal Agents , year =
-
[27]
arXiv preprint arXiv:2307.15043 , year=
Universal and transferable adversarial attacks on aligned language models , author=. arXiv preprint arXiv:2307.15043 , year=
-
[28]
Findings of the Association for Computational Linguistics: EACL 2024 , pages=
Do-not-answer: Evaluating safeguards in LLMs , author=. Findings of the Association for Computational Linguistics: EACL 2024 , pages=
2024
-
[29]
do anything now
" do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models , author=. Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security , pages=
2024
-
[30]
Advances in Neural Information Processing Systems , volume=
Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models , author=. Advances in Neural Information Processing Systems , volume=
-
[31]
arXiv preprint arXiv:2508.07407 , year=
A comprehensive survey of self-evolving ai agents: A new paradigm bridging foundation models and lifelong agentic systems , author=. arXiv preprint arXiv:2508.07407 , year=
-
[32]
arXiv preprint arXiv:2511.20857 , year=
Evo-memory: Benchmarking llm agent test-time learning with self-evolving memory , author=. arXiv preprint arXiv:2511.20857 , year=
-
[33]
Behavior Safety of Autonomous Interactive Agents: Risks, Attacks, Defenses and Evaluation , author=. Evaluation , volume=
-
[34]
Proceedings of the Twentieth European Conference on Computer Systems , pages=
Hybridflow: A flexible and efficient rlhf framework , author=. Proceedings of the Twentieth European Conference on Computer Systems , pages=
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.