REVIEW 2 major objections 6 minor 22 references
Gathered, Not Admitted: How Attention Brings a Latent Variable into Verbalizable Form
T0 review · 2 major / 6 minor · reviewed 2026-08-27 · deepseek-v4-flash
Pith's one-line read Attention gathers latent variables into readable form; there is no query-position gate.
desk verdict A careful, well-controlled test of the admission story for verbalizable workspaces; the demand-gated attention-gathering result holds up, but the title outruns the evidence by a step. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the Jacobian lens, a per-layer average linear operator $J_\ell = E[\partial h_{\text{final}}/\partial h_\ell]$ that reads each layer through the model's final unembedding; the readout is the percentile rank $R_z$ of the gold concept under the lens after the final normaliser. The mechanism that carries the claim is activation patching with separated patch depth and readout depth: patching a donor activation at layer $\ell$ and reading at a fixed distance above it distinguishes installation from survival, and substituting at mid-depth transports the donor's variable 17x or more above any shallower layer under a non-saturating readout. The paper's own inference labels the lower edge of that window a survival failure, with re-derivation from the still-present passage the favoured but unproven cause.
What would settle it
Mask the passage after an early patch at L33 and check whether the donor's value survives at the readout. The paper predicts it will not, because the lower edge is a survival failure with re-derivation from the passage as the likely cause; if the value survives with the passage masked, the lower edge is an artifact of the patch operation rather than a transport boundary.
Extended reading notes
Core claim
The central claim is that the verbalizable, causally substitutable form of a latent variable at the queried position is installed by attention-mediated transport within a mid-depth window (about L36–L45, fractional depth 0.56–0.66 on two architectures), not by unmasking a variable that was already there. A donor's value installed below the window fails to survive to the readout; above the window, substitution starts destroying rather than substituting. One shared linear map decodes the variable from every arm, including the control that needs it for nothing, and demand raises lens visibility beyond what applying an operator to a supplied value produces. The paper is explicit that this rules out admission at the query position, not a gate on the attention route itself, and that the gather is necessary but not sufficient.
Load-bearing premise
The whole depth-localisation argument rests on activation patching installing the donor's value faithfully at the patched layer, so that a null or weak result below the window is read as 'not transportable' rather than 'substitution failed'.
Editorial extensions
If this is right
- Interpretability tools that read lens visibility as evidence that a model is 'using' a concept need a causal check: a readout shift can be nearly identical for components whose behavioral effects differ by 7.4x.
- Workspace-style controllers that write into the late band or gate a workspace will not find a pre-existing query-position gate in these models; what demand changes is the amount of attention-mediated transport into the position.
- Causal localisation of latent variables should separate patch depth from readout depth; conflating them produces a spurious onset, as the paper found in its own early runs.
- The same fractional depth window appearing in a 64-layer hybrid and a 62-layer dense model suggests the gathering depth has a scale-free component across architectures.
Reading between the lines
- If the gather is demand-specific and mediated by attention, steering attention heads at fractional depth roughly 0.6 should shift how flexibly a model uses a latent quantity; this is a testable intervention the paper does not run.
- The sevenfold demand difference in gather amount suggests a training-based test: models trained with more tasks requiring flexible reuse should show a larger gather at that depth and perhaps an earlier survival boundary.
- The paper's calibration warning generalises beyond the Jacobian lens to any probing method that reads a linear direction and treats its magnitude as causal influence.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper formalizes an 'admission' account of verbalizable representations: a latent variable z is already present at the queried position at all depths, and task demand only controls whether a gate lets it be read. Using a Jacobian lens on Qwen3.6-27B and a benchmark whose five arms share an identical context but differ in whether z is needed, the paper reports: (i) lens visibility rises with demand against an accuracy-matched, format-matched control; (ii) a single linear probe decodes z in every arm, including the arm that needs z for nothing; (iii) activation-patching depth sweeps localize donor-substitutable content to a mid-depth window around L39, with the attention branch carrying most of the effect and no tested MLP output contributing positively inside the window; (iv) behavioral counterfactual and survival measurements give the window two measured edges, replicated at the same fractional depth on gemma-4-31B; and (v) a methodological caution that lens readout shifts are not calibrated measures of behavioral use. The paper concludes that the readable form is gathered by attention into the query position rather than admitted from a variable already sitting there.
Significance. If the central claim holds, the paper makes a substantive contribution to mechanistic interpretability: it separates availability from visibility, localizes the causal mechanism to attention-mediated transport, and provides a template of controls—label symmetry, paired bootstrap over semantic instances, a confirmation split for the selected layer and head, multiple donor pairings, adversarial destruction bounds, cross-architecture replication, and an artifact-level audit command. The paper is also unusually honest about measure saturation, selection provenance, and single-pairing fragility, and it reports negative and null results rather than only confirmatory ones. The main reservation is that the refutation of the admission account depends on activation-patching fidelity at shallow layers, an assumption the paper itself flags but does not independently test; this is the load-bearing point behind the recommendation below.
major comments (2)
- [§4, §5, and Appendix C] The refutation of an admission gate at the query position rests on the null transport results below L36, summarized as 'at least seventeen times less of the variable is transportable than inside a mid-depth window' and 'across the twelve layers immediately below that window no component transports it' (§4). This null is interpretable only if activation patching installs a donor value faithfully at shallow layers. Section 4 explicitly concedes the alternative: 'An early encoding that is nonlinear, lens-misaligned or destroyed by substitution would read as weak.' The behavioral counterfactual of §5 and the survival experiment of §5 use the same substitution operation (donor activations inserted at a layer and read later), so they do not independently rule out installation failure; a value that was never installed in a usable form would produce exactly the observed nulls and the 'value installed at L33 is entirely gone' result. Because an admission gate could act on such a non-substitutable early encoding, the paper's central claim is conditional on an untested patchability assumption. The authors should either add a patchability control (for example, patch a shallow layer and read out immediately at distance 0–1 with both the lens and a nonlinear probe, using a layer known to be behaviorally relevant as a positive control), or explicitly narrow the conclusion to 'no transportable, lens-readable variable is admitted at the query position' rather than 'no admission gate exists there.' As written, the title and abstract assert the stronger form.
- [§3, Table 1, and §10] The claim that 'demand raises a concept's lens visibility beyond what applying an operator to a supplied value produces' is not cleanly identified by the flexible−supplied contrast. In Table 1, supplied answers at accuracy 1.000 while flexible answers at 0.940, so the +0.0504 contrast is not accuracy-matched; Section 10 acknowledges this as a 'ceiling in the 2×2 that no analysis removes.' The problem is compounded because the operator-only contrast (supplied−control) changes sign with the readout and the interaction is significantly negative, so the decomposition of 'inference of z' from 'operator application' is not identified. Since the accuracy-matched flexible−control contrast is larger (+0.0891), the paper does not need the 'beyond supplied value' claim for its main visibility result, but the abstract and Contribution (1) present that contrast as a headline dissociation. Please either add an accuracy-matched supplied arm on at least the primary checkpoint, or rephrase the claim so that the operator-held-fixed result is presented as suggestive evidence subject to the ceiling, rather than as a measured 'beyond' effect.
minor comments (6)
- [§2.1 and Table 4] The text says 'we report every transport result under all three' readouts (Rz, Lz, Mz), but Table 4 and several appendix tables report only Rz and Lz; please either add Mz columns or clarify the sentence to say 'where available.'
- [§4] The phrase 'seventeen times less of the variable is transportable' is awkward; consider 'the transportable amount is at least seventeen times smaller' and state explicitly in the text that the denominator is the largest cell below the window taken within each resample, not a fixed cell.
- [Figure 1 and §10] The caption of Figure 1 states that the results 'do not rule out a gate acting on the gather itself,' but this important scope condition appears only in the caption and in §10; it should be stated once in §1 or §4 so that readers do not take the title to rule out all gates.
- [Table 6] For attn.L39.H15 under the leave-one-out mean ablation, the report-arm cost is exactly +0.000 with interval [+0.000,+0.000]; please state whether this is a rounding artifact of discrete accuracy counts or a genuinely zero effect, since a zero-width interval otherwise looks like a degenerate result.
- [§4 and Appendix D] The language-family head decomposition is single-pairing, as noted in §6, but §4 and Appendix D present the H15 numbers without a repeated cross-reference to that limitation; adding one sentence at each occurrence would prevent readers from over-weighting a single draw.
- [General] The five-arm notation (automatic, report, flexible, control, supplied) is used heavily; a small repeated definition in the captions of Figures 2, 3, and 9 would improve readability.
Circularity Check
No load-bearing circularity: the transport claim is independently measured with held-out instances, null controls, and a lens-free behavioural counterfactual.
full rationale
The derivation chain is self-contained and the headline negative result does not reduce to its own inputs. The admission hypothesis is explicitly constructed as the authors' formalisation ('This is our formalisation of the hypothesis, not a claim Gurnee et al. (2026) make'), and it is then tested against fresh measurements rather than assumed. The main transport result is not fitted into existence: L39 and H15 were selected on a 20/25-instance screen, frozen, and the paper reports a confirmation split showing the counterfactual and mediation hold on held-out instances ('most of every downstream result rests on instances the selection never saw'). The mediating direction is derived from the same Jacobian lens used for readout, which is a shared-measurement concern, but the paper does not stop there: the behavioural counterfactual is lens-free, four controls (random direction, rival-mean, worst rival, orthogonalised gold) run on the same pairs, and the readout-against-use comparison in §8 explicitly shows the lens readout is uncalibrated as a measure of causal influence, with a matched positive control in §7. The acknowledged limitation that an early nonlinear or lens-misaligned encoding would read as weak is a validity caveat about activation patching, not a circular reduction: the paper states the narrower claim and offers two measures with disjoint failure modes agreeing. No load-bearing step is equivalent to its inputs by construction; citations to Gurnee et al. (2026) provide the lens/J-space machinery but are not used to force the transport conclusion, and the authors distinguish their increment (demand-dependence over fixed context, installation/survival separation) from that prior work.
Assumptions & free parameters
free parameters (4)
- L39 gathering layer =
layer 39 of 64
- H15 head at L39 =
attention head 15, layer 39
- Readout distance for transport grid =
3 to 5 layers above patch
- Accuracy floor for interpretable cells =
task accuracy around 0.5, interpretable cells from 0.53 to 0.81
assumptions (3)
- domain assumption The Jacobian lens readout after the model's final normalizer is a faithful instrument for whether a layer's content is verbalizable; the depth-dependent rescaling gauge is the relevant invariance.
- domain assumption Activation patching installs the donor's content at the recipient layer in a semantically faithful way, so that null or weak effects below the window are interpretable as lack of transport or survival rather than as substitution failure.
- domain assumption The five benchmark arms differ only in task demand; label symmetry, shared context, and matched prompt format isolate the demand contrast.
Cite this review
Pith. "Pith review of Gathered, Not Admitted: How Attention Brings a Latent Variable into Verbalizable Form." pith.science (2026). https://pith.science/paper/FGYYMLQU
@misc{pith2026260815022,
author = {Pith},
title = {Pith review of: Gathered, Not Admitted: How Attention Brings a Latent Variable into Verbalizable Form},
year = {2026},
howpublished = {\url{https://pith.science/paper/FGYYMLQU}},
note = {Machine review of arXiv:2608.15022}
}
read the original abstract
Language models hold latent quantities in a form they can report on, and more of a quantity is present in that form when the task requires reusing it flexibly. What causes a representation to enter that form is open, and the word workspace invites an admission story: a gate that decides what gets in. Testing it on open-weight models with Jacobian lenses, over a benchmark whose five arms share an identical context, we find no gate where it predicts one. Demand raises a concept's lens visibility beyond what applying an operator to a supplied value produces: +0.050 [+0.045, +0.057] in percentile rank on our primary checkpoint, positive on all four we measure, though that arm answers at ceiling and the accuracymatched contrast is stronger under that readout. At the same time one shared linear map decodes the variable from every arm, the control included, at 6.4-9.0x its selection-corrected floor. What produces the later readable form at the queried position is attention-mediated gathering inside a mid-depth window: separating patch depth from readout depth puts transport there at least 17x above anywhere shallower under non-saturating readouts, with no tested MLP output contributing positively inside it. Under the saturating percentile rank the same grid does not localise the window, which is a fact about that measure. An arm that needs the variable for nothing concentrates sevenfold less, so the window is demand-specific. That window has two measured edges, a survival failure below and destruction above, and it falls at the same fractional depth in a 64-layer hybrid and a 62-layer dense model from another family. We localise where the variable is installed and read, not the route from the passage, which transports nothing. But the readout is not a calibrated measure of use: three components move it to within 12% of one another and differ 7.4x in what they do to the answer.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Eliciting Latent Predictions from Transformers with the Tuned Lens
Nora Belrose, Igor Ostrovsky, Lev McKinney, Zach Furman, Logan Smith, Danny Halawi, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens.arXiv preprint arXiv:2303.08112,
-
[5]
Dissecting Recall of Factual Associations in Auto-Regressive Language Models
arXiv:2304.14767. Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching.arXiv preprint arXiv:2304.05969,
-
[6]
Where Computation Lives Inside TabPFN: Causal Localisation of Attention Head Function
Aditya Gupta, Rahul Kumar, Sourav Mandal, and Nikhil Deshpande. Where computation lives inside tabpfn: Causal localisation of attention head function.arXiv preprint arXiv:2606.12917,
-
[7]
Verbalizable Representations Form a Global Workspace in Language Models
URLhttps:// transformer-circuits.pub/2026/workspace/index.html. arXiv:2607.15495. Diancheng Kang, Zheyuan Liu, Ningshan Ma, Yue Huang, Zhaoxuan Tan, and Meng Jiang. Prompt- activation duality: Improving activation steering via attention-level interventions.arXiv preprint arXiv:2605.10664,
work page Pith review arXiv 2026
-
[8]
Anis Khemais. A theory of conditional collapse under low-rank weight-space ablations.arXiv preprint arXiv:2608.03620,
-
[9]
arXiv:2311.17030. 10 Alex McKenzie, Keenan Pepper, Stijn Servaes, Martin Leitgab, Murat Cubuktepe, Mike Va- iana, Diogo de Lucena, Judd Rosenblatt, and Michael S. A. Graziano. Endogenous resistance to activation steering in language models.arXiv preprint arXiv:2602.06941,
- [11]
-
[12]
Roman Prosvirnin, Victor Minchenkov, Alexey Soldatov, and Vladimir Bashun. Silent alarm: A j-space protocol for comparing danger recognition across models and quantization levels.arXiv preprint arXiv:2607.12792,
Show all 22 references
-
[13]
Preprint, not peer reviewed
doi: 10.20944/preprints202607.0895.v1. Preprint, not peer reviewed. Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. Function vectors in large language models. InInternational Conference on Learning Representa- tions (ICLR),
-
[15]
Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West
arXiv:2211.00593. Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. Do llamas work in english? on the latent language of multilingual transformers. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pp. 15366–15394,
-
[16]
Junde Wu, Jiayuan Zhu, Fengling Liu, Minhao Hu, and Jiazhen Pan
arXiv:2402.10588. Junde Wu, Jiayuan Zhu, Fengling Liu, Minhao Hu, and Jiazhen Pan. J-cot: Chain-of-thought in j-space.arXiv preprint arXiv:2607.21981,
-
[17]
Spanish
arXiv:2309.16042. A THE BENCHMARK,AND THE INVARIANTS IT ENFORCES One instance, all five arms.Instancelang 000169, verbatim apart from the passage. Every arm receives the same prompt up to and including the blank line, then one instruction: Reference table: Finnish -> W Japanes...
-
[20]
true workspace
This run is single-pairing. Gurnee et al. (2026) note that the absence of verbalizable content in early layers may be realormay be lens degeneracy there, leaving open that a “true workspace” extends earlier. Our behavioural counterfactual never touches the lens and is also nul...
2026
-
[22]
Artifact names are given without their directory (one per experiment family under the data root) and without the model slug each file carries as a suffix
12span m* 80 pairs readout against behaviour, same trialstrial level80 pairs survival of an installed valuerepair span1 n100100 pairs repair source, 4 position modesrepair source v2 n100100 pairs necessity, leave-one-out ablationloo necessity50 instances/arm mediation (×3)wide...
2022
-
[24]
the instruction changed
On the single final token, identical across arms by construction, a control layer leads instead, so the effect is a property of the nine-token span and we report it as such. 17 Figure 4: What travels across checkpoints.(a, b)The distractor-controlled counterfactual against fra...
2018
-
[55]
Worst bound
resid.L39counterfactual+0.400 [+0.200,+0.600] +0.364 [+0.236,+0.491] mediation,resid.L39+0.120 [+0.000,+0.280] +0.218 [+0.109,+0.327] mediation,resid.L42+0.080 [+0.000,+0.200] +0.127 [+0.055,+0.218] The counterfactual holds on unseen instances, and the mediation isstrongerther...
2023
-
[80]
active at END
It is a ratio of two pairing-sensitive quantities, which is why it behaves worst of anything we measure; the smaller run’s L42 value sits inside its range and its L39 value above it, which is why the share is reported as a range and never as a point. D TRANSPORT AND THE ATTENT...
2023
-
[2022]
Andrew Nam, Henry Conklin, Yukang Yang, Thomas Griffiths, Jonathan Cohen, and Sarah-Jane Leslie
arXiv:2202.05262. Andrew Nam, Henry Conklin, Yukang Yang, Thomas Griffiths, Jonathan Cohen, and Sarah-Jane Leslie. Causal head gating: A framework for interpreting roles of attention heads in transformers. InAdvances in Neural Information Processing Systems (NeurIPS),
-
[2023]
When activation oracles learn not to read: Concept-specific blind spots in fine-tuned oracles.arXiv preprint arXiv:2607.23379,
Tobias Bersia and Tatiana Gaintseva. When activation oracles learn not to read: Concept-specific blind spots in fine-tuned oracles.arXiv preprint arXiv:2607.23379,
-
[2024]
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart M
arXiv:2310.15213. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart M. Shieber. Investigating gender bias in language models using causal mediation analysis. InAdvances in Neural Information Processing Systems,
-
[2025]
Gregory N
arXiv:2411.08745. Gregory N. Frank. How alignment routes: Localizing, scaling, and controlling policy circuits in language models.arXiv preprint arXiv:2604.04385,
-
[2026]
Learning to chain operations by routing information through a global workspace.arXiv preprint arXiv:2503.01906,
Hugo Chateau-Laurent and Rufin VanRullen. Learning to chain operations by routing information through a global workspace.arXiv preprint arXiv:2503.01906,
Reviewed August 27, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.