REVIEW 3 major objections 4 minor 28 references
Sparse MLP-neuron readouts can both detect and selectively correct LLM tool-use failures, cutting wrong calls by 80%.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-04 00:59 UTC pith:5BPXGGNB
load-bearing objection A serious, well-controlled empirical paper on sparse internal readouts for tool-use failures; the main caveat is that validity detection may partly read source signatures rather than failure-specific signals, but the steering results are protected by good controls. the 3 major comments →
A Few Neurons Reveal When LLMs Misuse Tools: Sparse Detection and Selective Steering for Reliable Tool Use
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that three tool-use failure modes—over-calling, missing, and invalid arguments—are linearly decodable from a small, failure-specific set of MLP neuron activations, and that the same contribution-selected neuron basis supports both sparse detection and causal steering. On six models across three model families, decision failures are detected before generation with ROC-AUC 0.90–1.00 and validity is detected from the generated call span with ROC-AUC 0.86–0.90, using as few as 1–2 neurons for missing, 2–16 for over-calling, and roughly 128 for validity. The sparse detectors match or outperform dense residual-stream probes while using 23–627 times fewer input features. When t
What carries the argument
The load-bearing object is the contribution-selected shared neuron basis. Each MLP neuron is scored by the norm of its residual-stream write (activation magnitude times down-projection column norm), aggregated over failure and correct rollouts; the top-k from each side form a basis that is simultaneously the feature space for an L1-regularized logistic detector and the anchor for per-layer correct-minus-failure steering directions. This is what allows the framework to answer 'when' (from the detector support) and 'how' (from the layerwise directions) using the same set of neurons, and what makes the loop closed rather than two unrelated probes.
Load-bearing premise
The whole loop assumes that a prompt whose eight natural rollouts consistently fail (or consistently succeed) is a faithful label of failure propensity; if the detector is instead keyed to surface cues—such as whether a tool is offered at all, or which source a call came from—the sparse-readout interpretation and the transferred steering would both be weakened.
What would settle it
Take prompts where the same prompt sometimes produces a tool call and sometimes abstains across rollouts (the excluded mixed-outcome cases) and ask whether the detector can separate those individual rollouts; if AUC is near chance, the detector separates only stable endpoints, not the failure itself. Alternatively, rephrase a stable-failure prompt without changing tool availability and check whether the detector still fires.
If this is right
- A monitor using only 1–2 neurons for missing and 2–16 for over-calling reaches ROC-AUC 0.90–1.00 before generation, so cheap pre-generation screening is feasible.
- The same sparse basis that detects failures yields layerwise steering directions; the correct-minus-failure direction suppresses unnecessary calls and elicits omitted ones, while layer-profile-matched random directions do not reproduce the effect.
- Gating these directions on detector risk removes the collateral damage of unconditional steering: pooled over-calling falls 80% and tool-required accuracy rises 14.2 percentage points across all six models.
- Under chain-of-thought reasoning, the over-calling signal migrates from the prompt boundary to the token before the closing reasoning tag, meaning monitors should be placed where the decision actually crystallizes.
- Validity is layer-flat and not steerable, so it remains a post-generation, pre-execution safety check; this bounds what activation steering can fix.
Where Pith is reading between the lines
- If the stable-endpoint labels are a fair proxy for real failure propensity, the same read-steer loop could be adapted to other agent errors—unsafe actions, refusals, hallucinated citations—since the method itself is not tool-specific.
- A deployment caveat the paper leaves implicit: because mixed-outcome prompts are discarded, the detector is only certified on clear-cut cases; in a live agent where the same prompt sometimes calls and sometimes abstains, the gate's reliability is untested.
- A testable next step is to convert the validity detector's block/flag decision into an automatic regeneration policy; the paper explicitly notes no repair mechanism was evaluated, so the validity AUC is a monitor, not a fix.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces PRISMS, a closed-loop framework for detecting and steering tool-use failures in LLMs. For each failure mode (over-calling, missing, validity), it builds a shared basis of contribution-selected MLP neurons, fits an L1-regularized logistic probe on signed activations, and constructs layerwise correct-minus-failure steering directions from the same basis. A detector gate applies the corresponding direction only on predicted failures. Across six models from Qwen3, Llama-3.1, and Gemma families, the paper reports pre-generation ROC-AUC 0.90–1.00 for over-calling and missing, call-span ROC-AUC 0.86–0.90 for validity, and pooled over-calling reduction from 0.131 to 0.026 with capability accuracy rising from 0.689 to 0.831 under gated steering. The paper includes prompt-disjoint splits, stable-endpoint labels, matched random direction/gate controls, and candid appendix limitations.
Significance. If the central claims hold, PRISMS would demonstrate that sparse, individually addressable MLP features can serve both as lightweight failure monitors and as selective control handles for tool use, with clear practical value. The paper is unusually strong on controls: held-out prompt-disjoint splits, matched random directions and gates, stable-endpoint audits, and explicit statements of what the sparse-support numbers do and do not mean. These controls make the behavioral steering results credible. The main risk is interpretational: the detectors may be reading scenario/source identity rather than failure-specific signals, and the paper's own exclusion of BFCL and its per-source validity numbers show that this risk is real. The gated-steering results are partly protected by the shuffled-gate and random-direction controls, but the headline 'failure-specific neuron' reading of the detection results is not yet established.
major comments (3)
- [Appendix A.3, C.7, Table S3] The validity detector's pooled AUC appears inflated by source identity. BFCL is excluded because near-zero error rate would let a probe rank source rather than validity, but the same concern applies to the three included sources, whose invalid rates differ sharply (e.g., Qwen3-4B: Glaive .102, ToolACE .492, xLAM .369, Table S3). The paper's own per-source numbers in C.7 (Glaive .905, xLAM .817, ToolACE .823 vs. pooled .886) show the pooled number lies above two of three within-source numbers. Please report source-controlled AUC (within-source training/evaluation, or source-balanced resampling) and a source-only baseline. Without this, the 'failure-specific' interpretation of the validity detector is not supported.
- [Eqs. S2–S4, Table S11] The pre-generation over-calling detector may be separating 'is a tool offered' from 'is this call erroneous.' For Qwen3-4B the baseline NTA rate is 0.534 while DT is 0.011, so the NTA/DT stratum itself is a near-perfect cue available at the final prompt token. The stable-endpoint filter (Eq. S4) removes mixed-outcome prompts, making the task easier. No text-only or prompt-feature baseline is reported to rule out these cues. Please add controls: e.g., stratum-balanced training, per-stratum AUC, and a bag-of-words/prompt-template classifier. This is load-bearing for the claim that the sparse readouts are failure-specific.
- [Appendix A.3, Eq. S4] The stable-endpoint definition excludes all mixed-outcome prompts from basis construction, probe fitting, and probe evaluation. The headline detection numbers therefore apply only to prompts that consistently fail or succeed across R=8 rollouts. A deployed monitor must score every prompt. Please report detection on the complete held-out set including mixed prompts, or quantify the excluded fraction and show that the detector's ranking on mixed prompts is not arbitrary. The paper discloses this limitation, but its impact on the central detection claim is not assessed.
minor comments (4)
- [Abstract / A.6] The abstract says 'only 1-2 MLP neurons for missing, 2-16 for over-calling, and approximately 128 for validity,' while A.6 clarifies these are unconstrained sparse-support limits and the operational detector supports are tens to hundreds. Please make this distinction in the abstract or main text to avoid over-reading.
- [Eq. (5), Table S11] The 80% pooled over-calling reduction is a prompt-count-weighted average across models; please state explicitly in the main text that this is a macro average over the six models, since per-model OC reductions vary widely (e.g., Gemma barely changes).
- [Table 2 / Table S10] The 'K=1' rows report union counts that are not simply K times the number of layers; please clarify the exact construction so readers can reproduce the budget.
- [C.7] Per-source validity comparison is reported only for Qwen3-4B; please add per-source numbers for the other models or explicitly state that this analysis is limited to one representative model.
Circularity Check
No significant circularity: empirical fitting with held-out evaluation, validation-locked operating points, and matched controls; the one author self-citation is not load-bearing.
full rationale
The paper's central claims are empirical rather than derivational. Detectors are fit on training prompts and evaluated on prompt-disjoint test prompts; thresholds and steering strengths are selected on validation and frozen before test (Sec. 3, 4.1; Appendix A.7). Steering directions are constructed from development corpora and tested against matched-random, label-shuffled, shuffled-gate, and random-vector controls; the paper explicitly disclaims that sparse detection alone establishes causal neuron specialness (Sec. 3.1, 3.3, 4.1). Stable-endpoint labels (Eq. S4) and the BFCL exclusion (Appendix A.3) raise legitimate construct-validity concerns about source/scenario cues, but those are confounds, not circularity: the fitted label is not definitionally equal to the predicted quantity, and the evaluation remains on held-out prompts. The only author self-citation (Zhao, Ke, and Huang 2026) appears in related work on safety knowledge neurons and is not load-bearing for the tool-use monitoring or steering claims. No derivation step reduces by construction to its inputs.
Axiom & Free-Parameter Ledger
free parameters (6)
- basis budget k=m=ceil(0.03N) =
3% of layer–neuron pairs
- L1 regularization C =
C in {0.01, 0.03, 0.1, 0.3}, selected on validation ROC-AUC
- detector threshold tau_f =
0.7 (locked across all six models)
- steering strength alpha_f =
0.2 (locked across all six models)
- intervention token window =
final prompt token + first five generated tokens
- rollout counts R =
R=8 for decision failures, R=16 for validity, temperature 0.7
axioms (8)
- domain assumption FFN output is a weighted sum of value vectors, and contribution c=|a|*||v|| identifies behaviorally relevant neurons.
- domain assumption Tool-use failures are linearly separable in signed, layer-normalized MLP write features.
- domain assumption Natural-rollout labels from AST/normalized-value matching and stable-endpoint aggregation are accurate and deployment-relevant ground truth.
- ad hoc to paper Excluding mixed-outcome prompts from decision-basis construction, probe fitting, and probe evaluation does not bias the central claims.
- ad hoc to paper BFCL can be excluded from pooled validity scoring because its near-zero error rate would let a probe rank source signatures.
- domain assumption Adding a scaled unit direction to the MLP output before the residual stream changes tool-call behavior causally and does not simply corrupt generation.
- domain assumption Validation-selected thresholds and strengths generalize across models and held-out prompts.
- domain assumption Per-rollout validity calls can be treated as independent units for ROC-AUC/AP.
read the original abstract
Agentic LLMs exhibit three consequential tool-use failures: invalid arguments (validity), unnecessary calls (over-calling), and omitted calls when tools are needed (missing). We find that a small, failure-specific set of MLP neurons could distinguish such failures with linearly separable decision boundaries. Building on this observation, we introduce PRISMS (Probing Representations In Support of Monitoring and Steering), a closed-loop framework that shares a failure-specific neuron basis between sparse detection and activation steering. PRISMS selects contribution-critical MLP neurons and fits an L1-regularized detector on their activations. Across six models from the Qwen3, Llama, and Gemma families, over-calling and missing are detected at the pre-generation prompt boundary with ROC-AUC 0.90-1.00, while validity is detected from the generated tool-call span with ROC-AUC 0.86-0.90. These results are achieved with highly sparse readouts: only 1-2 MLP neurons for missing, 2-16 for over-calling, and approximately 128 for validity. These sparse detectors match or outperform dense residual-stream baselines using 23-627 times fewer features. The shared neuron basis also supports bidirectional control over tool-calling behavior, suppressing unnecessary calls and eliciting omitted ones. PRISMS therefore gates intervention on predicted failure risk to mitigate the collateral effects of unconditional steering. Across all six models, PRISMS reduces pooled over-calling rate by 80% (from 0.131 to 0.026) while increasing tool-required accuracy by 14.2 percentage points (from 0.689 to 0.831). PRISMS thus provides lightweight failure detection and selective intervention across model families.
Figures
Reference graph
Works this paper leans on
-
[1]
and Mao, Huanzhi and Yan, Fanjia and Ji, Charlie Cheng-Jie and Suresh, Vishnu and Stoica, Ion and Gonzalez, Joseph E
Patil, Shishir G. and Mao, Huanzhi and Yan, Fanjia and Ji, Charlie Cheng-Jie and Suresh, Vishnu and Stoica, Ion and Gonzalez, Joseph E. , booktitle =. The Berkeley Function Calling Leaderboard (. 2025 , publisher =
2025
-
[2]
Liu, Weiwen and Huang, Xu and Zeng, Xingshan and Hao, Xinlong and Yu, Shuai and Li, Dexun and Wang, Shuai and Gan, Weinan and Liu, Zhengying and Yu, Yuanqing and Wang, Zezhong and Wang, Yuxian and Ning, Wu and Hou, Yutai and Wang, Bin and Wu, Chuhan and Wang, Xinzhi and Liu, Yong and Wang, Yasheng and Tang, Duyu and Tu, Dandan and Shang, Lifeng and Jiang,...
2025
-
[3]
Zhang, Jianguo and Lan, Tian and Zhu, Ming and Liu, Zuxin and Hoang, Thai and Kokane, Shirley and Yao, Weiran and Tan, Juntao and Liu, Zhiwei and Feng, Yihao and Niebles, Juan Carlos and Heinecke, Shelby and Wang, Huan and Savarese, Silvio and Xiong, Caiming , booktitle =. x. 2025 , publisher =. doi:10.18653/v1/2025.naacl-long.578 , url =
-
[4]
2024 , howpublished =
2024
-
[5]
Proceedings of the 42nd International Conference on Machine Learning , series =
Reducing Tool Hallucination via Reliability Alignment , author =. Proceedings of the 42nd International Conference on Machine Learning , series =. 2025 , publisher =
2025
-
[6]
The Reasoning Trap: How Enhancing
Yin, Chenlong and Sha, Zeyang and Cui, Shiwen and Meng, Changhua and Li, Zechao , booktitle =. The Reasoning Trap: How Enhancing. 2026 , publisher =. doi:10.18653/v1/2026.acl-long.376 , url =
-
[7]
Hao, Yupu and Jin, Zhuoran and Liao, Huanxuan and Liu, Kang and Zhao, Jun , booktitle =. Pushing the Limits of. 2026 , publisher =. doi:10.18653/v1/2026.findings-acl.710 , url =
-
[8]
Azaria, Amos and Mitchell, Tom , booktitle =. The Internal State of an. 2023 , publisher =. doi:10.18653/v1/2023.findings-emnlp.68 , url =
-
[9]
AAAI 2026 Workshop on Trustworthy Agentic AI , year =
Internal Representations as Indicators of Hallucinations in Agent Tool Selection , author =. AAAI 2026 Workshop on Trustworthy Agentic AI , year =
2026
-
[10]
2026 , url =
Sun, Chung-En and Liu, Linbo and Yan, Ge and Wang, Zimo and Weng, Tsui-Wei , journal =. 2026 , url =
2026
-
[11]
Yao, Shunyu and Zhao, Jeffrey and Yu, Dian and Du, Nan and Shafran, Izhak and Narasimhan, Karthik and Cao, Yuan , booktitle =
-
[12]
Advances in Neural Information Processing Systems , volume =
Reflexion: Language Agents with Verbal Reinforcement Learning , author =. Advances in Neural Information Processing Systems , volume =
-
[13]
arXiv preprint arXiv:2605.07990 , year =
Tool Calling Is Linearly Readable and Steerable in Language Models , author =. arXiv preprint arXiv:2605.07990 , year =
-
[14]
Model-Adaptive Tool Necessity Reveals the Knowing--Doing Gap in
Cheng, Yize and Fan, Chenrui and JafariRaviz, Mahdi and Rezaei, Keivan and Feizi, Soheil , journal =. Model-Adaptive Tool Necessity Reveals the Knowing--Doing Gap in. 2026 , url =
2026
-
[15]
The Twelfth International Conference on Learning Representations , year =
A Simple and Effective Pruning Approach for Large Language Models , author =. The Twelfth International Conference on Learning Representations , year =
-
[16]
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages =
Transformer Feed-Forward Layers Are Key-Value Memories , author =. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages =. 2021 , publisher =. doi:10.18653/v1/2021.emnlp-main.446 , url =
-
[17]
Proceedings of the 41st International Conference on Machine Learning , series =
Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications , author =. Proceedings of the 41st International Conference on Machine Learning , series =. 2024 , publisher =
2024
-
[18]
2025 , url =
Gao, Cheng and Chen, Huimin and Xiao, Chaojun and Chen, Zhiyi and Liu, Zhiyuan and Sun, Maosong , journal =. 2025 , url =
2025
-
[19]
Zhao, Chongwen and Ke, Yutong and Huang, Kaizhu , booktitle =. Unraveling. 2026 , publisher =. doi:10.18653/v1/2026.eacl-long.83 , url =
-
[20]
arXiv preprint arXiv:2605.08513 , year =
A Single Neuron Is Sufficient to Bypass Safety Alignment in Large Language Models , author =. arXiv preprint arXiv:2605.08513 , year =
-
[21]
Advances in Neural Information Processing Systems , volume =
Inference-Time Intervention: Eliciting Truthful Answers from a Language Model , author =. Advances in Neural Information Processing Systems , volume =. 2023 , url =
2023
-
[22]
arXiv preprint arXiv:2308.10248 , year =
Steering Language Models with Activation Engineering , author =. arXiv preprint arXiv:2308.10248 , year =
-
[23]
and Wang, Zifan and Mallen, Alex and Basart, Steven and Koyejo, Sanmi and Song, Dawn and Fredrikson, Matt and Kolter, J
Zou, Andy and Phan, Long and Chen, Sarah and Campbell, James and Guo, Phillip and Ren, Richard and Pan, Alexander and Yin, Xuwang and Mazeika, Mantas and Dombrowski, Ann-Kathrin and Goel, Shashwat and Li, Nathaniel and Byun, Michael J. and Wang, Zifan and Mallen, Alex and Basart, Steven and Koyejo, Sanmi and Song, Dawn and Fredrikson, Matt and Kolter, J. ...
2023
-
[24]
2026 , url =
Sui, Yuan and Chen, Yulin and Li, Yibo and Jiang, Xue and He, Yufei and Dong, Yihong and He, Xiaoxin and Gao, Tianyu and Hooi, Bryan , journal =. 2026 , url =
2026
-
[25]
2026 , url =
Wang, Youjin and Zhou, Run and Ma, Yingjie and Fu, Rong and Liang, Jiani and Cao, Shuaishuai and Huang, Min and Fang, Tao and Pan, Liangming , journal =. 2026 , url =
2026
-
[26]
arXiv preprint arXiv:2607.05790 , year =
Controlling Tool Use with Heading-Specific Activation Steering , author =. arXiv preprint arXiv:2607.05790 , year =
-
[27]
Transactions of the Association for Computational Linguistics , volume =
Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals , author =. Transactions of the Association for Computational Linguistics , volume =. 2021 , publisher =. doi:10.1162/tacl_a_00359 , url =
-
[28]
Advances in Neural Information Processing Systems , volume =
Adversarial Examples Are Not Bugs, They Are Features , author =. Advances in Neural Information Processing Systems , volume =. 2019 , url =
2019
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.