{"id":"998bc32a-da0e-4b15-a57d-09f8aead9084","arxiv_id":"2608.22339","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Skill memories that record both the procedure and the boundary conditions for when not to use it outperform procedure-only memory on tool-use LLM benchmarks and reverse a newly measured wrong-tool imitation trap.","lead":"A study of tool-using language agents finds that storing more skill memories from past successes can make the agent more likely to pick the wrong tool when a new task only looks similar. Adding explicit notes about when a skill does and does not apply improves success rates and reduces safety failures across three benchmark environments.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"BASM's gains hinge on semantically correct extracted boundary fields, but the paper validates only their format, not their truth; an oracle audit of f_ext is needed before the mechanism claim is assured.","rationale":"The reader's weakest_assumption identifies exactly this issue: no validation that f_ext's boundary fields are accurate, complete, or faithful to the environment's true dynamics. I agree, and this is the most load-bearing point in the paper. The Skill Imitation Trap diagnosis and the BASM remedy both depend on boundary fields being semantically real: the probe analysis shows the model attends to boundary spans and that masking that attention restores imitation, but attention to a false or generic boundary statement could also shift logits through instruction-following. The deterministic validator ψ in Eq. (7) checks only schema and formatting, not truth. The paper even calls its construction 'precision-first' without reporting any precision measurement. The artifact and limitation sections state that code and derived skill memories are not yet released, so the audit cannot be performed from the manuscript alone. This is not a reason to reject: the internal controls, especially equal-token padding and procedure-span knockout, are sound and partially rule out length and formatting confounds. But the external validity of the method rests on an unverified extractor. A concrete oracle audit of extracted fields, plus an oracle-field replacement experiment, would settle whether the concern lands. Since the reader already arrived at a conditional verdict, my read does not change that verdict; it sharpens the condition under which acceptance should be granted.","tokens_in":17106,"tokens_out":4955,"duration_ms":50505,"concrete_test":"Take 100 trajectories from AppWorld and 100 from BFCL, held out from all skill-library construction and from the main evaluation. For each decision state, use the environment's simulator and tool schemas to compute oracle labels: whether each candidate tool is applicable or safe, what should be avoided, and what recovery action is correct. Run f_ext on the associated successful and failure contexts, then compare each extracted field (A/R/N/F) against the oracle labels using exact matching plus LLM-judged semantic matching, reporting per-field precision and recall. Then rerun BASM on the same cases with extracted fields replaced by oracle fields; if oracle-field BASM materially exceeds extracted-field BASM, semantic error is the bottleneck. If precision is high and oracle replacement does not improve performance, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism and all downstream benchmark gains depend on the boundary fields A, R, N, F produced by f_ext in Eq. (6). The paper validates these fields only with the deterministic operator ψ, which enforces JSON structure, required slots, non-empty lists, and tool-schema validity (Sec. 4.2); it does not check whether the fields are semantically true of the environment. A skill could pass ψ while stating an applicability condition that does not hold, a risk cue that is false, or an avoidance rule that happens to match the test instance's expected answer. The same-prompt knockout (Sec. 3.4) demonstrates that decision-token attention to the boundary span shifts logits, but it cannot distinguish 'true boundary information changes the decision' from 'the model follows boundary-shaped instructions regardless of their truth.' Since BASM's advantage is claimed to come from validity-scoped transfer rather than extra context, the semantic accuracy of f_ext is load-bearing. The manuscript itself emphasizes 'precision-first construction' but reports no measured precision of extraction; no human audit, no oracle comparison, and no released library are provided. If f_ext's boundary fields are often inaccurate or task-specific, the reported gains may not transfer to new environments, even though all internal controls (length, formatting, attention knockout) behave as reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper argues that success-distilled skill memory in LLM agents can be harmful in states that resemble past successes but require different tools, a failure it calls the Skill Imitation Trap. Through logit-margin probes, attention analysis, and same-prompt attention knockout on Qwen3 models, the authors report that retrieving more procedural skills increases wrong-tool preference, while adding boundary fields (applicability, risk, avoidance, recovery) reverses this trend and is causally tied to boundary-span attention. The proposed BASM pipeline constructs a seven-slot skill library offline, retrieves and formats skills under a token budget, and adds a runtime boundary checker and repair loop. Evaluations on BFCL, AppWorld, and AgentDojo across 8B–397B models report gains over success-distilled baselines and a memory-free baseline, with ablation studies separating the contributions of boundary prompting and runtime repair.","tokens_in":17395,"tokens_out":8084,"duration_ms":69350,"significance":"If the Skill Imitation Trap is real, it challenges the monotonic-improvement assumption common in skill-memory self-evolution and gives a concrete failure mode to guard against. The probe design is a notable strength: it includes equal-length padding controls, a same-prompt attention knockout, procedure-span and random knockouts, and a negative linear-probe control, making the mechanism claim more credible than a purely benchmark-driven paper. The benchmark coverage across three environments and four model scales is also valuable. However, the lack of any validation of the semantic accuracy of the LLM-extracted boundary fields, the absence of variance estimates for benchmark tables, and a few inconsistent headline numbers leave the transferability and the strength of the causal claim less certain than the abstract suggests.","major_comments":[{"comment":"The validation operator ψ enforces only structural requirements (JSON parsing, required fields, list-valued slots, non-empty risk/avoidance lists, tool-schema validity), so the semantic correctness of the boundary fields A, R, N, F is never measured. The paper claims that BASM transfers because skills are \"validity-scoped records,\" but if f_ext hallucinates or misstates applicability conditions, the reported gains could arise from arbitrary instruction-following rather than from true boundary semantics. I request a human audit or oracle comparison on a sample of extracted skills (e.g., 100 skills per benchmark) reporting precision of each boundary field, and an error analysis showing how boundary-inaccurate skills affect the benchmark results.","section":"§4.2, Eq. (6)"},{"comment":"All benchmark tables appear to report single runs without variance estimates or significance tests. On the largest model, Qwen3.5-397B-A17B, BASM's BFCL overall is 53.75 versus 53.38 for Proc. Skill, a 0.37-point difference, and AppWorld success is 54.76 versus 52.98; these differences are likely within run-to-run noise. The abstract's \"consistently outperforms\" needs support from repeated runs (at least on the key comparisons), confidence intervals, or a significance test across instances.","section":"§5.1–5.2, Tables 1–4"},{"comment":"The repeated claim that boundary knockout \"recovers 69.4% of the imitation-trap effect\" is not reproducible from the reported numbers. At k=2 the Proc. Skill wrong-tool margin is 10.71 and BASM is 2.26; the knockout raise is +3.45. That is 40.8% of the Proc. Skill-to-BASM gap; relative to the Proc. Skill increase from k=0 (7.31) to k=2 (10.71), it is about 101%; relative to BASM's suppression from k=0 to k=2 (7.31 to 2.26), it is about 68.3%. Please state the denominator used to compute the 69.4% figure and label it consistently in the text and figure captions.","section":"§3.4, Fig. 6 vs Fig. 2"},{"comment":"The advertised \"reduces attack success rate by 4.6%\" is not supported by Table 3: on Qwen3-8B the AgentDojo ASR drops from 8.01 to 5.90 (2.11 points, about 26% relative), and on Qwen3-14B from 3.79 to 2.95 (0.84 points, about 22% relative). Specify whether the 4.6% is a different aggregate, an absolute value, or relative to a specific baseline; otherwise the headline number is unexplained.","section":"Abstract; Table 3"}],"minor_comments":[{"comment":"The hyperparameters λ_sem, λ_tool, λ_step, γ, β, and k are not specified; report their values and a sensitivity analysis, since the retrieval and formatting pipeline depends on them.","section":"§5.1, Eq. (8), Eq. (10)"},{"comment":"The padding controls also show an increase from Tok to Twrong (neutral padding 0.10 to 1.05; procedure padding 0.19 to 0.82), so the statement that controls \"do not reproduce this pattern\" should be softened to note that the controls do not reproduce the peak at Trepair or the magnitude of the BASM shift.","section":"§3.3, Fig. 3"},{"comment":"The boundary checker's use of \"LLM-assisted judgment\" for ambiguous natural-language conditions is underspecified; describe the criterion or prompt used, otherwise the runtime control is not fully reproducible.","section":"§4.2, Eq. (11)"},{"comment":"The Limitations section does not acknowledge the two main methodological gaps: the absence of semantic validation of f_ext and the single-run nature of the benchmark tables; both should be explicitly listed as limitations or addressed in the experiments.","section":"Limitations"}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read it. First, the paper finds a real, counterintuitive failure mode: when an agent retrieves success-distilled skills, more retrieved skills can raise, not lower, its preference for a semantically similar but wrong tool. The probe is clean: wrong-tool margin rises from 7.31 to 10.71 at k=2, and Proc. Skill underperforms the memory-free base on BFCL in the benchmark tables, so the trap shows up outside the lab. Second, the fix mostly works. Boundary fields (applicability, risk, avoidance, recovery) push the margin down to 2.26, the same-prompt attention knockout recovers 69.4% of the imitation effect, and the ablation study gives the two components (boundary prompting, runtime repair) distinct and complementary credit.\n\nWhat is actually new: prior work (Wang et al. 2026a) had noted over-imitation as an empirical signal, but this paper supplies the quantitative mechanics—state-conditioned boundary attention, wrong-tool logit suppression, knockout control—and a practical schema that converts a skill into a state-conditioned record. The options-framework lineage is acknowledged. Credit where due: the controls are serious. Equal-length padding, procedure padding, random and procedure knockout, residual-stream linear probe, and paired equal-token comparisons rule out the cheap explanations. That is honest experimental work.\n\nSoft spots, in order. The biggest one is the semantic fidelity of the extractor f_ext in Eq. 6. The validation operator ψ checks JSON structure and slot presence, not whether the extracted applicability conditions and risk cues are true of the environment. The same-prompt knockout shows that the decision token reads the boundary span and that reading changes logits, but it cannot distinguish “true boundary information changes the decision” from “the model follows boundary-shaped instructions regardless of their truth.” Since the paper claims validity-scoped transfer, that distinction matters. No human audit, no oracle comparison, no released skill library. That is the load-bearing joint, and it is currently unexamined.\n\nSmaller issues: benchmark tables are single runs with no error bars; the BFCL gain on the largest model is 0.37 points; and several free parameters (lambda weights, gamma, beta, k) are left under-specified. The absence of released code and data makes the variance problem harder to check. None of these sink the central argument as stated.\n\nWho this is for: anyone building skill-memory or retrieval-augmented tool-use agents. The mechanism analysis is worth a reading group on its own. My recommendation: send it to peer review, and let the reviewers require (a) artifact release, (b) variance or significance reporting, and (c) an audit of f_ext—even a small human-annotated or oracle-based precision check on extracted boundary fields. With that audit, the claim would be much more convincing; without it, the transfer claim is plausible but not nailed down.","headline":"BASM is a real fix for a real failure mode in skill-memory agents; the mechanism evidence is strong, but the unvalidated semantic accuracy of the extracted boundary fields is the one load-bearing joint.","tokens_in":17921,"tokens_out":2553,"duration_ms":23787},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"More skill memory can push LLM agents toward wrong tools.","keywords":["skill memory","tool-use LLM agents","Skill Imitation Trap","boundary-aware skill memory","retrieval-augmented generation","agent self-evolution","function calling","prompt injection safety"],"falsifier":"An experiment that would settle the claim: on a held-out set of inapplicable-state tasks, scramble each skill's boundary fields (or replace them with another skill's fields) and remeasure the wrong-tool margin. If BASM's suppression persists under scrambled boundaries, the effect is not semantic boundary information; if it collapses, the extractor's fidelity becomes the decisive factor.","tokens_in":16943,"feed_emoji":"🛠️","tokens_out":6971,"duration_ms":57866,"temperature":0.7,"pith_summary":"This paper argues that the standard recipe for skill-memory agents—distill successful trajectories into procedures and retrieve them when relevant—has a hidden failure mode. When a task resembles a past success but actually needs a different tool, the more retrieved skills the model sees, the more confidently it reaches for the wrong tool; the paper measures a 47% increase in wrong-tool margin over a memory-free agent. To fix this, the authors propose Boundary-Aware Skill Memory (BASM), which adds four boundary fields to each skill—applicability conditions, risk cues, avoidance rules, and recovery notes—so retrieved skills act as state-conditioned guidance rather than unconditional templates. Across BFCL, AppWorld, and AgentDojo at four model scales, BASM improves task success and safety relative to success-distilled memory baselines, and attention-knockout probes locate the mechanism in decision-token attention to boundary text.","feed_headline":"More skill memory can push LLM agents toward wrong tools","feed_subtitle":"Adding applicability, risk, avoidance, and recovery cues reverses the effect and boosts task success and safety.","key_machinery":"The load-bearing object is the seven-slot skill schema s=(g, π, T, A, R, N, F)—goal, procedure, tools, applicability conditions, risk cues, avoidance rules, and recovery notes—where the last four slots form the boundary component B(s). At inference, BASM retrieves skills with a multi-view score, always exposes the procedural core, and only includes boundary text when a boundary-relevance score exceeds a threshold, under a fixed token budget. A boundary checker G suppresses proposed calls that violate avoidance rules or applicability conditions, and a repair module injects recovery hints after local failures. The mechanism's causal role is established by attention knockout: removing decision-token attention to the boundary span reopens the imitation trap, so the boundary fields function as a state-conditioned anti-imitation channel.","core_discovery":"The central claim is that success-distilled skill memory is not monotonically beneficial: on decision states where the retrieved skill is semantically similar but inapplicable, wrong-tool preference grows with retrieval depth (from a wrong-tool margin of 7.31 at k=0 to 10.71 at k=2 for procedural memory), while BASM reverses the curve, dropping the margin to 2.26. The paper further claims that this reversal is causal and mechanistically specific: attention from the decision token shifts toward boundary fields in risky and repair states, and same-prompt knockout of boundary-span attention restores 69.4% of the imitation-trap effect. On benchmark tasks, these probes translate into up to 23.8% higher AppWorld success, up to 5.0% higher BFCL accuracy, and a 4.6-point lower AgentDojo attack success rate than success-distilled baselines, with fewer interactions on average.","pith_inferences":["If boundary fields can be generated reliably, the same schema should extend to dynamically changing tool APIs, where static applicability conditions must be rechecked against the current schema.","The attention-head specialization found in the probes suggests a testable extension: steering or fine-tuning the identified boundary-reading heads could yield the same suppression at lower context cost.","A natural stress test is to corrupt or invert boundary fields: if the method's advantage collapses under semantically plausible but wrong boundaries, then extraction quality is the true bottleneck for deployment.","The Skill Imitation Trap may generalize beyond tools to any retrieved exemplar, such as few-shot demonstrations or workflow memories, wherever semantic similarity masks conditional inapplicability."],"forward_implications":["Skill-memory systems should stop treating retrieved success as positive transfer; adding explicit validity boundaries is a necessary component, not an optional annotation.","In single-turn function-calling benchmarks, boundary-conditioned prompting alone accounts for most of the gain; the runtime repair loop adds value mainly in stateful, multi-step environments.","Avoidance and recovery fields carry safety value: removing avoidance rules raises attack success rate under prompt-injection attacks, and removing recovery notes lowers utility across benchmarks.","Retrieval depth should be budgeted: indiscriminate retrieval amplifies wrong-tool preference, so prompt formatters should expose boundary evidence selectively rather than stacking procedures.","The attention-knockout result implies that the benefit is not simply extra context, and that similar boundary annotations could be tested for other in-context memory formats."],"supporting_citations":[{"why":"Introduced the Voyager-style skill library distilled from successful trajectories, which serves as the baseline paradigm BASM targets.","marker":"Wang et al., 2023"},{"why":"ExpeL supplies the success-distilled verbal-lesson memory approach that motivates the positive-transfer assumption BASM challenges.","marker":"Zhao et al., 2024"},{"why":"Formalized skill applicability through options-framework initiation sets, the conceptual precedent for BASM's boundary-conditioned reuse.","marker":"Konidaris et al., 2014"},{"why":"Provides BFCL v3, whose miss_func and miss_param subsets directly instantiate the Skill Imitation Trap.","marker":"Patil et al., 2025"},{"why":"Provides AppWorld, the stateful multi-step benchmark where BASM's runtime repair loop is evaluated.","marker":"Trivedi et al., 2024"},{"why":"Provides AgentDojo, the prompt-injection benchmark used to measure utility and attack success rate.","marker":"Debenedetti et al., 2024"},{"why":"Supplies the retrieval-augmented generation machinery on which BASM's skill retrieval and reranking are built.","marker":"Lewis et al., 2020"}],"fun_headline_variants":["Skill memory can backfire in LLM tool choice","Boundary-aware skills stop LLM imitation trap","Add boundaries, avoid wrong tools: BASM for LLMs","When not to imitate: skill memory with guardrails","Risk cues in skill memory cut LLM tool errors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's gains rest on the LLM-based skill extractor's boundary fields being semantically accurate: it is assumed that the generated applicability, risk, avoidance, and recovery statements match the environment's true dynamics, but the validation operator only checks their format and presence, not their truth.","fun_headline_variants_meta":{"raw":{"variants":["Skill memory can backfire in LLM tool choice","Boundary-aware skills stop LLM imitation trap","Add boundaries, avoid wrong tools: BASM for LLMs","When not to imitate: skill memory with guardrails","Risk cues in skill memory cut LLM tool errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000288,"raw_usage":{"total_tokens":1728,"prompt_tokens":1022,"completion_tokens":706,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":638,"completion_tokens_details":{"reasoning_tokens":629}},"tokens_in":638,"tokens_out":706,"duration_ms":7018,"temperature":1.0,"reasoning_tokens":629,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-27T18:31:11.243188+00:00","model_set":{"reader":"deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.0-alltime-deepseek-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"An experiment that would settle the claim: on a held-out set of inapplicable-state tasks, scramble each skill's boundary fields (or replace them with another skill's fields) and remeasure the wrong-tool margin. If BASM's suppression persists under scrambled boundaries, the effect is not semantic boundary information; if it collapses, the extractor's fidelity becomes the decisive factor.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}