Pith. sign in

REVIEW 4 major objections 14 references

Keeping pretrained FFN weight directions fixed and gating only activation amplitudes at inference improves tool-structured LLM outputs when a sample-level gate decides when to accept the change.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-15 08:56 UTC pith:2LCGRNAW

load-bearing objection Careful offline FFN amplitude gating for tool/JSON routes, with honest negative boundaries; useful engineering signal that still needs online validation. the 4 major comments →

arxiv 2607.11183 v2 pith:2LCGRNAW submitted 2026-07-13 cs.CL

Amplitude-Only FFN Intervention for Tool-Structured LLM Inference Method: Gated Evaluation Protocol, and Cross-Model Empirical Results

Sheng Xu , Junhua Wang , Boyuan Huang , Ke Jia , Jiadun Zhu , Zhen Chen This is my paper
classification cs.CL
keywords Amplitude GatingFFN interventiontool-structured inferenceSwiGLUinference-time optimizationlearned gatesfunction callingJSON structured output
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Tool-using language models often fail on small format, argument, or function-call errors even when the underlying intent is right. This paper argues that those last-mile failures can be reduced at inference time by intervening inside the feed-forward network without retraining weights. After showing that direction-changing residual projection often harms more than it helps, the authors propose Amplitude Gating: keep the pretrained FFN matrices untouched and only modulate activation magnitudes at carefully chosen SwiGLU sites, then accept the modified path only when online-observable features predict net benefit. Under a protocol that separates oracle headroom from fixed configs and learned gates, the method yields category-level gains on Qwen3.5-9B tool routes and large Hermes JSON gains on Qwen3-8B, while Qwen2.5-7B keeps oracle headroom without a working category gate. The practical claim is therefore narrow and useful: tool-structured inference is the credible first target, and deployment needs model- and category-specific routing plus baseline fallback.

Core claim

Amplitude Gating—non-destructive modulation of FFN activation magnitudes while preserving pretrained weight directions—contains useful candidate repairs for tool-structured generation; when paired with learned gates that use only online-observable features, it produces positive offline gains on Qwen3.5-9B tool/function-call routes (+4.27 pp category, about +7.6 pp on Hermes function-call) and on Qwen3-8B Hermes JSON mode (+11.36 pp), whereas the same candidate space on Qwen2.5-7B is not yet captured by current gates, so safe use requires model- and category-specific routing rather than a universal switch.

What carries the argument

Amplitude Gating (AG): at selected SwiGLU FFN sites (P1/P2/P3 and refined P1s/P2a/P2b), mask or shrink activation dimensions by energy or entropy-mass rules (including Newton-Schulz windowed ranking) while leaving FFN weights fixed; a gate using prefix/format, early logprob trajectory, uncertainty, and divergence/energy features accepts the AG path or falls back to baseline, with evaluation that separates combination-oracle headroom from fixed and learned-gate results.

Load-bearing premise

Offline learned gates built from features the authors treat as online-observable will keep the same fix-versus-harm profile once the system is run live, even though no prospective online A/B or time-split validation is reported and the same feature families fail on one of the three models.

What would settle it

A held-out prospective online deployment (or time-split) of the same category gates on Qwen3.5-9B tool routes and Qwen3-8B JSON routes that shows zero or negative net fix/harm relative to baseline would falsify the deployable-route claim; likewise, showing that the reported lifts disappear under a denser e-grid or independent seeds would undermine the current fixed-probe evidence.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Production systems that need reliable tool calls or JSON can treat gated FFN amplitude intervention as a first inference-time optimization surface before weight updates.
  • Deployments must route by model and task category (function-call vs JSON vs long QA) rather than enable one global AG switch.
  • Hidden-dimensional sites such as P3 and P1s are preferred first production targets over expensive expansion-dimensional P2/P2a/P2b points.
  • Non-NS and Newton-Schulz-windowed AG should both remain available candidate families; a gate chooses between sharp current-token and short-window trajectory scoring.
  • Oracle headroom without a positive learned gate (as on Qwen2.5-7B) is not a green light for shipping.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the gate’s early-trajectory and format features are the real carriers of lift, similar amplitude gates may transfer to other constrained decoders (schema-guided agents, multi-step planners) without new FFN taxonomies.
  • The ORP-to-AG pivot suggests a design rule for internal interventions generally: preserve learned geometry and only modulate scale when sample-level risk is observable.
  • Failure of shared feature families on Qwen2.5-7B implies that wider FFN expansions may increase candidate diversity while making online selection harder—motivating expansion-aware gate features.
  • Training-time use of the same FFN trajectory signals to clean tool-supervision data, as the authors flag as future work, could shrink the offline oracle–learned gap before any online A/B.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 0 minor

Summary. The paper studies inference-time feed-forward intervention for tool-structured LLM outputs. After an internal Orthogonal Residual Projection (ORP) attempt that changed residual/FFN directions and often produced more harm than fix, the authors propose Amplitude Gating (AG): keep pretrained FFN weight directions fixed and modulate only activation magnitudes at SwiGLU sites (P1/P2/P3 and refined P1s/P2a/P2b), optionally with Newton–Schulz windowed ranking. A central methodological contribution is an evaluation protocol that separates combination-oracle headroom from best fixed configurations and learned gates restricted to online-observable features, with sample-level accounting and task-aware strict vs partial10 metrics. Across Qwen3.5-9B (hybrid attention), Qwen3-8B, and Qwen2.5-7B, AG is weakly positive in aggregate and strongest on tool/JSON/function-call routes: e.g., Qwen3.5-9B tool category 38.66%→42.92% (+4.27 pp) under a category mixed gate, Hermes function-call ~+7.6 pp; Qwen3-8B Hermes JSON +11.36 pp; Qwen2.5-7B shows oracle headroom without a valid category gate. The authors conclude that tool-structured inference is the credible first target and that deployment requires model- and category-specific routing, with prospective online validation still required.

Significance. If the offline learned-gate lifts hold under nested held-out training and later online validation, the work would establish a conservative, architecture-general FFN intervention surface for tool-using agents—distinct from weight updates, representation steering, and decoder-level constrained decoding. Strengths that should be credited include: (i) an explicit pivot from a failed direction-changing method (ORP) to a non-destructive amplitude-only design; (ii) a leakage-aware protocol that refuses to treat combination oracles or pooled multi-config accounting as deployable evidence; (iii) sample-level fix/harm and partial10 diagnostics; (iv) multi-checkpoint coverage spanning hybrid and dense attention; and (v) bootstrap CIs on feature-family ablations that both support Qwen3/Qwen3.5 tool routes and honestly mark Qwen2.5-7B as a boundary case. The contribution is primarily empirical and methodological rather than theoretical; its value for the field depends on whether the reported gates are truly out-of-sample.

major comments (4)
  1. §5.1–5.2 and §6.7: The central deployable claim rests on category-level mixed/ridge learned gates (e.g., Qwen3.5-9B tool 38.66%→42.92%, Hermes FC ~+7.6 pp; Qwen3-8B Hermes JSON +11.36 pp). Section 6.7 reports bootstrap CIs on a stable hash split for feature-family ablations, but the main Section 5 headline numbers do not clearly document a nested protocol in which gate model class, feature subset, and thresholds are frozen on a train split before any evaluation on the samples that produce those lifts. With ~1k gate-policy features and a 24-candidate AG matrix, this is load-bearing: without nested held-out (or time-split) gate training, the “deployable-feature-constrained” gains may shrink. Please state the exact train/eval split for every reported learned gate, freeze hyperparameters on train only, and re-report category lifts with CIs under that protocol.
  2. §4.2 and §5: Retained-mass values e∈{0.93,0.95} are fixed from earlier ORP/early-AG sweeps rather than re-selected on the reported test labels—good—but the paper still reports many “best fixed” and “best learned” cells over positions, families (non-NS vs NS), and datasets. For fixed-configuration claims, clarify whether the winning P/e/family cell was chosen with a held-out selection rule or by inspecting the same evaluation tables. If the latter, demote best-fixed results to exploratory and elevate only pre-specified single-route results (or report selection-corrected estimates).
  3. §2 and §5: Related work correctly distinguishes AG from constrained decoding / schema-guided generation, but there is no empirical baseline that applies a strong decoder-level JSON/tool-schema constraint (or constrained decoding + baseline) on the same Hermes JSON and ToolBench routes. Without that comparison, it is hard to know whether FFN amplitude intervention adds value beyond last-mile syntax enforcement—the failure mode the introduction emphasizes. A modest controlled comparison on Hermes JSON and at least one ToolBench split would substantially strengthen (or bound) the deployment claim.
  4. §5.3, §6.7, and §8: Qwen2.5-7B’s +4.03 pp tool oracle headroom with non-positive learned gates is an important negative result, but the manuscript still frames AG as cross-model “existence” of the effect. Given that all three checkpoints are Qwen-family SwiGLU models, the architecture-generality claim should be tightened to “cross-checkpoint within one family” in the abstract and conclusion until a non-Qwen family is tested, or the authors should add at least one external-family checkpoint under the same protocol.

Circularity Check

1 steps flagged

No load-bearing circularity: AG claims are empirical baseline comparisons on external tool/JSON benchmarks; internal ORP is motivational only.

specific steps
  1. self citation load bearing [§3.2–3.3 ORP archive; Ref. [5] project-archive note]
    "ORP was an internal exploratory method rather than an established external baseline. ... Reference [5] is retained as project-archive context, not as public or peer-reviewed evidence for the AG claims."

    Positions P1/P2/P3, e-sensitivity, and the direction-change failure mode are justified from the authors' own incomplete ORP archive and an internal DeepSeek-V4 note. This is self-citation, but it is not load-bearing for the AG performance claims: those rest on external-benchmark baseline deltas under the gated protocol, and the paper explicitly demotes [5] and incomplete ORP runs to diagnostics rather than proof of AG lifts.

full rationale

This is an empirical inference-time methods paper, not a first-principles derivation. The central claims (category learned-gate lifts on Qwen3.5-9B tool routes and Qwen3-8B Hermes JSON; Qwen2.5-7B as a negative boundary) are measured against external datasets (ToolBench, Hermes function-call/JSON, ConvFinQA, Leval, HEB RAG, MC suites) with explicit baseline pairing. Combination oracle is defined as label-aware upper bound and is excluded from deployable claims; learned gates are restricted to features the authors mark as online-observable, with label-derived fields forbidden. The e∈{0.93,0.95} grid is pre-specified from earlier sweeps rather than re-optimized on the reported test labels, and negative results (Qwen2.5-7B category gates, weak MC/choice gates) are reported rather than suppressed. Internal ORP and project-archive notes motivate positions and the amplitude-only pivot but do not substitute for the AG performance evidence. Incomplete nested documentation of gate train/test separation is a validity/overfitting concern, not a by-construction reduction of prediction to input. Score 1 only for non-load-bearing internal archive self-reference; no Eq./definition forces the headline lifts.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 4 invented entities

The central empirical claim rests on a small set of design choices (amplitude-only intervention, fixed e-grid, decode-only application, online-feature gate families) and the domain assumption that pretrained FFN directions should not be rotated at inference. Free parameters are the retained-mass e values, NS window, and gate thresholds/feature weights learned offline. Invented entities are project-specific methods and site labels rather than physical objects; independent external evidence for AG as a general principle is limited to the paper’s own Qwen runs.

free parameters (4)
  • retained entropy/energy mass e = 0.93 and 0.95 (paired grid)
    Main AG mask prefix uses e as retained mass; text/tool matrix fixes e∈{0.93,0.95} from earlier ORP/AG sweeps rather than learning per dataset. Results and fix/harm signs change with e at fixed positions.
  • Newton-Schulz window size w = w4
    NS AG ranks dimensions using a short recent-token window; experiments use w4 as a fixed design point without a window-size sweep.
  • learned gate thresholds and feature weights = model- and category-specific (offline)
    Mixed/ridge/single-indicator gates are fit on mining features; example ToolBench audit cites thresholds such as crossppl ≤ 1.25833. These determine accept/reject and thus reported lifts.
  • top-k dimension budget k / local-norm multiplier
    Taxonomy includes top-k and local-norm variants that introduce explicit keep budgets or std multipliers (archive notation still uses e for some of these).
axioms (5)
  • domain assumption Pretrained FFN weight directions encode semantic priors that should not be overwritten by inference-time direction-changing projection.
    Design pivot after ORP diagnostics (Section 3.3–3.4); motivates amplitude-only AG. Supported by ORP harm>fix runs but not proved as a general law.
  • domain assumption FFN blocks are a sufficiently architecture-general intervention surface across Transformer-like LLMs even when attention layouts differ.
    Stated in Introduction and Method; evidence limited to Qwen hybrid and dense SwiGLU checkpoints.
  • ad hoc to paper Deployable claims require online-observable features only; post-fork diagnostics and labels are forbidden in main gates.
    Core of the evaluation protocol (Sections 4.4–4.7); defines what counts as a positive result in this paper.
  • ad hoc to paper Decode-only intervention (prefill unchanged) is the correct safety/deployment boundary for AG.
    Section 3.8; explains weak MC effects and scopes where multi-token trajectories exist.
  • standard math Standard Transformer/SwiGLU forward computation and residual-stream structure as in cited Qwen reports.
    Background architecture assumptions used to define P-sites.
invented entities (4)
  • Amplitude Gating (AG) and entropy/NS AG variant family no independent evidence
    purpose: Non-destructive inference-time candidate generation by modulating FFN activation magnitudes under a gate-and-fallback policy.
    Central method; independent evidence is the paper’s own offline Qwen experiments, not external replications.
  • Orthogonal Residual Projection (ORP) no independent evidence
    purpose: Internal direction-changing precursor used as diagnostic failure mode motivating AG.
    Project-specific; not an established external baseline. Used to justify non-destructive design.
  • SwiGLU intervention taxonomy P1/P2/P3 and P1s/P2a/P2b no independent evidence
    purpose: Name and prioritize FFN input, gate-branch, value-branch, fused, and output amplitude sites.
    Implementation-specific coordinate system; refined points only partially mechanistically validated in the paper.
  • Combination-oracle vs fixed-config vs learned-gate evaluation protocol no independent evidence
    purpose: Prevent overstatement from pooled multi-config and best-selected accounting.
    Methodological construct of the paper; valuable but not independently standardized outside this work.

pith-pipeline@v1.1.0-grok45 · 30653 in / 4465 out tokens · 46179 ms · 2026-07-15T08:56:34.414938+00:00 · methodology

0 comments
read the original abstract

Large language models increasingly operate as tool-using agents, where small format, argument, or function-call errors can invalidate otherwise plausible responses. We study inference-time feed-forward network (FFN) intervention for improving structured outputs without retraining model weights. Our project began with Orthogonal Residual Projection (ORP), a direction-changing repair attempt that revealed sensitive SwiGLU FFN intervention sites but often caused more harm than fixes. We therefore propose Amplitude Gating (AG), a non-destructive alternative that preserves pretrained FFN weight directions and modulates only activation magnitudes during generation. We define a fine-grained intervention system spanning P1/P2/P3 and branch-specific P1s/P2a/P2b sites, and introduce an evaluation protocol that separates combination-oracle headroom from fixed configurations and learned gates, enforces sample-level accounting, and uses task-aware metrics for binary and partial-credit datasets. Across Qwen3.5-9B, Qwen3-8B, and Qwen2.5-7B, AG is weakly positive in aggregate but strongest on tool-structured tasks. On Qwen3.5-9B, a category-level learned gate improves tool/structured/agentic performance from 38.66% to 42.92% (+4.27 percentage points), with Hermes function-call tasks reaching about +7.6 points. On Qwen3-8B, Hermes JSON mode improves by +11.36 points. Qwen2.5-7B retains oracle headroom but current learned gates fail to capture it, showing that deployment requires model- and category-specific routing. Comparisons of entropy AG with Newton-Schulz-windowed AG show that neither family is uniformly dominant. These results identify tool-structured inference as the most credible first target for safe FFN-level inference optimization, while prospective online validation and broader cross-model evaluation remain necessary.

Figures

Figures reproduced from arXiv: 2607.11183 by Boyuan Huang, Jiadun Zhu, Junhua Wang, Ke Jia, Sheng Xu, Zhen Chen.

Figure 1
Figure 1. Figure 1: ORP failure diagnostic: P1 energy sweeps are non-monotonic, and the loaded SVD-ORP checkpoint produces more regressions [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: AG versus ORP design pivot: ORP changes SVD-derived projection directions, while AG keeps pretrained FFN weights fixed and [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: AG intervention points inside a Transformer FFN block. AG keeps FFN weights fixed and gates activation amplitudes at selected [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Matched e-grid design: e0.93 and e0.95 are paired probes inside the sensitive retained-mass region, so the experiment tests [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Qwen3.5-9B tool-structured results: baseline, learned gate, and oracle score by route. [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Qwen3-8B tool-structured results: baseline, learned gate, and oracle score by route. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Qwen2.5-7B boundary result: baseline, learned/current, and oracle score show headroom without a valid route. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Position-by-e sensitivity: matched Qwen3.5-9B pure-entropy slices show that both the FFN position and e-value change the sign [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: MC/choice diagnostics define the claim boundary: static P/e signals are small, and the current choice/trace feature gate is not [PITH_FULL_IMAGE:figures/full_fig_p021_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Task-category learned lift matrix: tool-structured routes are the most consistent positive region. [PITH_FULL_IMAGE:figures/full_fig_p022_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Cross-model routing: the strongest deployable route differs by checkpoint. [PITH_FULL_IMAGE:figures/full_fig_p023_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Oracle headroom versus deployable gain: large candidate-space headroom does not imply safe online capture. [PITH_FULL_IMAGE:figures/full_fig_p024_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Feature-family ablation with bootstrap confidence intervals: Qwen3/Qwen3.5 support tool-route gates, while Qwen2.5 remains a [PITH_FULL_IMAGE:figures/full_fig_p024_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Gate trade-off curve: stricter gates reduce harm exposure while also giving up fix capture. [PITH_FULL_IMAGE:figures/full_fig_p025_15.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

14 extracted references · 13 linked inside Pith

  1. [1]

    N., Kaiser, L., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention Is All You Need. NeurIPS. https://arxiv.org/abs/1706.03762

    Pith/arXiv arXiv 2017

  2. [2]

    Shazeer, N. (2020). GLU Variants Improve Transformer. arXiv:2002.05202. https://arxiv.org/abs/2002.05202

    Pith/arXiv arXiv 2020

  3. [3]

    Yang, A., Li, A., Yang, B., et al. (2025). Qwen3 Technical Report. arXiv:2505.09388. https://arxiv.org/abs/2505.09388

    Pith/arXiv arXiv 2025

  4. [4]

    Yang, A., Yang, B., Zhang, B., et al. (2024). Qwen2.5 Technical Report. arXiv:2412.15115. https://arxiv.org/abs/2412.15115

    Pith/arXiv arXiv 2024

  5. [5]

    DeepSeek-AI. (2026). DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. Project-archive technical note; retained as internal context, not as public or peer-reviewed evidence

    2026

  6. [6]

    Qin, Y., Liang, S., Ye, Y., et al. (2023). ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. arXiv:2307.16789. https://arxiv.org/abs/2307.16789

    Pith/arXiv arXiv 2023

  7. [7]

    Qu, C., Dai, S., Wei, X., Cai, H., Wang, S., Yin, D., Xu, J., and Wen, J.-R. (2024). Tool Learning with Large Language Models: A Survey. arXiv:2405.17935. https://arxiv.org/abs/2405.17935

    Pith/arXiv arXiv 2024

  8. [8]

    Cobbe, K., Kosaraju, V., Bavarian, M., et al. (2021). Training Verifiers to Solve Math Word Problems. arXiv:2110.14168. https://arxiv.org/abs/2110.14168

    Pith/arXiv arXiv 2021

  9. [9]

    Wang, Y., Ma, X., Zhang, G., et al. (2024). MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. arXiv:2406.01574. https://arxiv.org/abs/2406.01574

    Pith/arXiv arXiv 2024

  10. [10]

    Huang, Y., Bai, Y., Zhu, Z., et al. (2023). C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models. arXiv:2305.08322. https://arxiv.org/abs/2305.08322

    Pith/arXiv arXiv 2023

  11. [11]

    Phan, L., Gatti, A., Han, Z., et al. (2025). Humanity's Last Exam. arXiv:2501.14249. https://arxiv.org/abs/2501.14249

    Pith/arXiv arXiv 2025

  12. [12]

    Belcak, P., and Wattenhofer, R. (2023). Fast Feedforward Networks. arXiv:2308.14711. https://arxiv.org/abs/2308.14711

    Pith/arXiv arXiv 2023

  13. [13]

    Zou, A., Phan, L., Wang, J., et al. (2023). Representation Engineering: A Top-Down Approach to AI Transparency. arXiv:2310.01405. https://arxiv.org/abs/2310.01405

    Pith/arXiv arXiv 2023

  14. [14]

    T., and Louf, R

    Willard, B. T., and Louf, R. (2023). Efficient Guided Generation for Large Language Models. arXiv:2307.09702. https://arxiv.org/abs/2307.09702

    Pith/arXiv arXiv 2023