Pith. sign in

REVIEW 2 major objections 5 minor 11 references

Language models represent what you meant; they often fail to act on it.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 01:16 UTC pith:4YBTQGUD

load-bearing objection Careful probe-and-steer work that makes sender intent a real object; the early-representation to late-handle link is under-specified but the paper is honest about it. the 2 major comments →

arxiv 2607.03598 v1 pith:4YBTQGUD submitted 2026-07-03 cs.CL cs.AIcs.LG

They Infer What You Meant: Models Represent Communicative Intent More Reliably Than They Act On It

Alex Kwon This is my paper
classification cs.CL cs.AIcs.LG
keywords communicative intentlinear probingactivation steeringreadout lagpragmaticsGricean intentresidual streamlanguage models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

When you share something with a language model, it often answers the surface of the message and misses what you were doing by sending it: a finished project gets unsolicited critique, a late-night vent gets a risk assessment. This paper treats the sender's communicative intent as a first-class object of study and shows the failure is usually not ignorance. Across six open models in four families, a linear probe cleanly recovers the intent (recognize versus evaluate) from the model's default hidden states, even when the intent is never stated and must be inferred from context, and even in base pretraining checkpoints. Within a model the intent is decodable several layers before it drives the output; across models, whether the default reply honors it is model-specific. Where the gap is open, steering a direction closely tied to that representation recovers the intended behavior as well as an explicit instruction does, with no prompt, and the direction is near-orthogonal to a generic feedback knob. The practical upshot is a reframe: the model often already has what you meant; the fragile part is whether the readout routes it.

Core claim

A sender's communicative intent is robustly represented in the residual stream of language models and can be read out linearly, surface-independently, from pretraining onward, including when the intent must be pragmatically inferred; the common failure is a lagging or model-specific readout that does not act on that representation. Where default behavior discards the intent, a discriminative direction at a searched late layer is a causal handle that recovers honoring.

What carries the argument

The represent-then-lagging-readout chain on a surface-matched recognize-versus-evaluate contrast: leave-one-phrasing-out linear probes on default-pass hidden states, depth localization of decodability versus steerability, and causal recovery by adding the discriminative (logistic-weight) intent direction at a per-model late layer.

Load-bearing premise

That the steering direction is closely enough tied to the early-decoded representation that recovery counts as routing what was already represented, even though difference-of-means at the probe peak fails and the paper never isolates whether the later layer or the logistic direction is what makes the handle work.

What would settle it

On the three discard models, show that no direction extracted from the peak-probe layer (or a controlled factorial of logistic versus difference-of-means at peak versus late layers) can recover recognize-honoring without collapsing coherence or failing the specificity and near-orthogonality controls; or show that default honoring is already at ceiling once stimuli fully block request-detection and valence confounds.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper treats a sender's communicative intent (primarily recognize vs evaluate) as a first-class linear feature of language-model residual streams. A linear probe decodes that intent surface-independently from default-pass last-token activations across six models and four families, including base (pre-instruct) checkpoints and cases where intent must be pragmatically inferred; bag-of-words under leave-one-phrasing-out CV is at chance. Default behavior discards the intent on three of six models (unsolicited feedback on recognize shares). Where the gap is open, the logistic-weight direction at a per-model searched late layer is a causal handle: steering recovers honoring with a clean dose-response, matches an explicit intent prompt, is near-orthogonal to a feedback-behavior axis, and beats matched-norm random and difference-of-means controls. Depth sweeps show the probe saturates several layers before steering recovers. A second, lexically clean inferred axis (support vs help) is represented but does not yield a specific causal handle. Controls, human validation of the honoring measure, and nulls (including an inconclusive pre-registered geometry test) are reported throughout.

Significance. If the result holds, it reframes a common deployment failure (models answering surface content rather than what the sender was doing) as a readout problem on top of a robust, pretraining-learned representation, and it introduces a sender's goal—distinct from character ToM beliefs or user attributes—as a linear, steerable interpretability object. Strengths that raise the contribution above a standard probe-and-steer paper include: surface-matched identical suffixes with leave-one-phrasing-out and object-held-out CV; request-matched and valence-disentanglement construct controls; base-checkpoint results; human rater validation (Fleiss κ=0.76) of the behavioral measure; specificity against random, shuffled-label, difference-of-means, and opener-unembedding directions; near-orthogonality to the feedback axis; honest scoping of the support/help null and the inconclusive pre-registered geometry test; and released code/stimuli with a CPU-reproducible core chain. The represent-versus-readout decomposition with depth localization is a useful template for other pragmatic features.

major comments (2)
  1. [Section 6; Section 10 / Table 3] Section 6 (and the localization in Section 10 / Table 3): the central claim that steering 'routes a represented intent' rests on a direction that is only 'closely tied' to the early probe feature. Difference-of-means at the peak-probe layer (24) fails the sanity gate and can worsen discard; recovery requires the logistic-weight direction at a later searched layer (30 on Qwen-3B). The paper correctly flags that the logistic-at-24 / DoM-at-30 factorial is untested, so it is unknown whether the effective handle is the early-decodable intent feature or a later, possibly distinct computation that merely correlates with the labels. Without either the missing factorial cells or a geometric link (e.g., cosine between the peak-probe direction and the validated steer direction, or transfer of the early direction to the late layer), 'closely tied to the representation' and 'routes a represented int
  2. [Section 7 / Table 2] Section 7 / Table 2: for four of the six models the steer layer and coefficient are selected in-sample on the evaluation items; only Qwen-3B and Llama-8B receive a nested object-split validation. The paper flags this, but the non-overlapping recovery claims for Qwen-7B (and the ceiling claims for the other three) are therefore weaker than for the two split models. Either run the nested split for the remaining discard model(s) or demote those rows more clearly to exploratory status so the across-model stratification is not over-read.
minor comments (5)
  1. [Section 10] Section 10: the paper notes that probe-before-steer is common even for acted-on features and that a matched non-discarded control is missing. Consider elevating this caveat one notch in the main text (not only the localization section), since the depth gap is used as part of the represent-then-lag narrative.
  2. [Section 9; Appendix F] Section 9 / Appendix F: on Qwen-3B the body-reorientation test is inconclusive at the 100-token budget because separation dilutes; the refutation there rests on near-orthogonality alone. State this more prominently when claiming the handle is not opener-token biasing across all three discard models.
  3. [Section 8; Abstract] Section 8 / Appendix J: support-vs-help is correctly scoped to representation only after the specificity control fails. A one-sentence pointer in the abstract or contributions that the causal half of the story is established only on recognize/evaluate (and vent/solve) would prevent over-reading the third axis.
  4. [Figure 1; Table 1; Appendix K] Figure 1 and Table 1: bag-of-words is reported as 0.48 in the figure caption and main text but the per-model table (Appendix K) shows 0.46–0.48; keep a single consistent value or note the range.
  5. [Appendix E; Appendix M] Reproducibility: the Modal scripts and GitHub link are welcome; ensure the pre-registration document for the inconclusive geometry test (Appendix E) is in the release so the decision rule can be audited.

Circularity Check

0 steps flagged

No significant circularity: empirical probe-and-steer results with held-out CV, bag-of-words baselines, and independent behavioral scoring (lexicon/embedding/humans); mild under-specification of the causal handle is a validity gap, not a definitional reduction.

full rationale

This is standard activation-probe-and-steer work, not a first-principles derivation. Intent labels are fixed by the surface-matched stimulus construction (prefix sets recognize/evaluate; suffix identical), but the probe is evaluated under leave-one-phrasing-out and leave-one-object-out GroupKFold with an empirical permutation ceiling and bag-of-words at chance (0.48), so accuracy is not forced by surface. Steering success is scored by a separate feedback-offer lexicon, an independent sentence-embedding classifier, and three-way human majority vote (κ≈0.74–0.76), none of which re-read the probe accuracy or the logistic weights. Dose-response, random/shuffled-norm controls, near-orthogonality to the feedback axis, and nested object-split validation for layer/coefficient on two models further separate the intervention from the fit. The paper itself flags the untested logistic-at-24 / DoM-at-30 factorial and reports nulls (support/help specificity failure, inconclusive pre-registered geometry). The single self-citation (Kwon 2026) appears only in Related Work as a parallel dissociation and is not load-bearing. No equation reduces to its inputs by construction; the represent-then-lagging-readout chain is an empirical pattern, not a tautology. Score 1 reflects only the ordinary mildness that the searched late discriminative direction is called 'closely tied' without the missing factorial cells.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 1 invented entities

The work is experimental, not axiomatic. Load-bearing choices are design assumptions (binary intent contrast, surface-matched templates, lexicon/embedding honoring scores, linear residual-stream steering) and free parameters chosen by search (steer layer, coefficient α). No new physical entities are postulated. The central claim rests on these operational definitions plus standard linear-representation and residual-stream intervention assumptions from the interpretability literature.

free parameters (4)
  • steer layer (per model) = e.g. Qwen-3B L30, Qwen-7B L16, Llama-8B L19
    Chosen by sweep; not the peak-probe layer. Only Qwen-3B and Llama-8B validated with nested object split; other four selected in-sample (Section 7, Table 2).
  • steering coefficient α = ~0.5–1.0 (model-dependent)
    Dose-response on Qwen-3B; recovery dose ~1.0 times mean activation norm; usable range bounded by coherence collapse above 1.0 (Appendix C).
  • probe PCA components and logistic C = ≤40 PCA dims, C=1.0
    PCA ≤40 components, ℓ2 logistic C=1.0; standard but still pipeline hyperparameters that affect reported accuracy (Appendix B).
  • honoring / feedback-offer lexicon threshold = lexicon-based binary score
    Primary behavioral measure is a feedback-offer lexicon; validated against humans and embeddings but still a discrete scoring rule that defines the effect size.
axioms (4)
  • domain assumption Communicative intent of interest is well-approximated by a binary linear feature in residual-stream activations (recognize vs evaluate, and secondary axes).
    Standard linear representation hypothesis in mech-interp; invoked throughout Sections 3–6.
  • domain assumption Adding a unit direction scaled by α times mean activation norm at one decoder layer is a valid causal intervention on readout without destroying generation coherence at moderate dose.
    Standard activation-steering assumption (Turner, Rimsky, Zou); Section 6 and Appendix C.
  • ad hoc to paper Surface-matched identical suffixes plus leave-one-phrasing-out CV isolate intent from lexical surface at the probed token.
    Core experimental design (Section 2–3, Appendix A); bag-of-words at chance is the empirical check.
  • ad hoc to paper Default honoring of recognize-intent is a valid operationalization of whether the model 'acts on' represented intent.
    Section 5; human majority vote and embedding cross-check support the measure but do not make it the only possible pragmatic success criterion.
invented entities (1)
  • sender communicative-intent direction (recognize/evaluate discriminative residual-stream feature) independent evidence
    purpose: Serves as the linear, steerable object claimed to encode Gricean what-was-meant and to causally route honoring behavior.
    Not a new particle/force, but a postulated linear feature. Independent evidence is partial: decodable under held-out phrasings/objects, present in base models, transfers to hand-written messages, and steering moves behavior with specificity controls. Still defined relative to this paper's contrast set.

pith-pipeline@v1.1.0-grok45 · 23570 in / 3745 out tokens · 31875 ms · 2026-07-12T01:16:58.939648+00:00 · methodology

0 comments
read the original abstract

When a person shares something with a language model, the model often answers the surface of the message rather than what the sender was doing by sending it: share a finished project and it critiques the code; share a raw late-night line and it runs a wellness check. We treat the sender's communicative intent, the Gricean what-was-meant, as a first-class interpretability object, and show the failure is one of readout on top of a robust representation. A linear probe decodes the sender's intent, whether they want a thing recognized or evaluated, from a model's default-pass hidden states, cleanly and surface-independently, across six models and four families and in the base checkpoints. The representation generalizes further, to intent that is only pragmatically inferred, and to a second, lexically clean intent (support versus help). The behavioral half of the story, and every causal test, is established on the recognize/evaluate contrast, where what varies is whether the default output acts on the intent. The readout lags the representation in depth within a model (the intent is decodable several layers before it drives the output); across models, which ones act on it by default is model-specific, an observed stratification (three of six show the failure) that we do not read as a scaling law. Where the gap is open, a direction closely tied to the representation, the discriminative direction at a searched-for layer, is a causal handle: steering it recovers the intended behavior, as well as an explicit instruction does and with no prompt at all. This direction is near-orthogonal to the feedback-offering axis, so it routes a represented intent rather than a generic feedback knob, though at the recovery dose the routed intent can override an explicit request. We support each link with controls against obvious deflations and report the nulls as plainly as the confirmations.

Figures

Figures reproduced from arXiv: 2607.03598 by Alex Kwon.

Figure 1
Figure 1. Figure 1: The represent-then-lagging-readout chain, exemplified on Qwen-3B (a model where the readout gap is open). (1) A linear probe decodes the intent (recognize vs evaluate) from the default-pass hidden state at 1.00, while bag-of-words on held-out phrasings is at chance (0.48). (2) The default reply nonetheless offers unsolicited feedback, honoring a recognize-intent only about 0.65 of the time. (3) Steering th… view at source ↗
Figure 2
Figure 2. Figure 2: The effect requires the learned direction. Distribution of the behavior separation S = feedback(toward-evaluate)−feedback(toward-recognize) for 48 random directions of matched norm (gray) and 12 shuffled-label directions (blue), against the true intent direction (red line). On both models the true direction sits beyond the entire control distribution; no random or shuffled direction reaches it (permutation… view at source ↗
Figure 3
Figure 3. Figure 3: Represented before routed. Per layer, how decodable the intent is (blue, probe accuracy) and how much steering at that layer recovers recognize-intent honoring (red, with bootstrap 95% CIs), against the default baseline (dotted). On both models the probe saturates at layers where steering has not yet lifted honoring above baseline; recovery arrives only deeper (late and sharp for Qwen-3B, earlier and more … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

11 extracted references · 7 linked inside Pith

  1. [1]

    Guillaume Alain and Yoshua Bengio

    URLhttps://arxiv.org/abs/2404.14219. Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. InInternational Conference on Learning Representations (ICLR), Workshop Track,

    Pith/arXiv arXiv

  2. [2]

    Language models as agent models

    Jacob Andreas. Language models as agent models. InFindings of the Association for Computational Linguistics: EMNLP 2022,

    2022

  3. [3]

    URL https: //aclanthology.org/2022.findings-emnlp.423

    doi: 10.18653/v1/2022.findings-emnlp.423. URL https: //aclanthology.org/2022.findings-emnlp.423. Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances.Computational Linguistics, 48(1):207–219,

    doi:10.18653/v1/2022.findings-emnlp.423 2022

  4. [5]

    Matteo Bortoletto, Constantin Ruhdorfer, Lei Shi, and Andreas Bulling

    URLhttps://arxiv.org/abs/2303.08112. Matteo Bortoletto, Constantin Ruhdorfer, Lei Shi, and Andreas Bulling. Benchmarking mental state representations in language models. InICML 2024 Workshop on Mechanistic Interpretability,

    Pith/arXiv arXiv 2024

  5. [6]

    Albert Q

    URLhttps://arxiv.org/abs/2410.13648. Albert Q. Jiang et al. Mistral 7b,

    arXiv

  6. [7]

    Alex Kwon

    URLhttps://arxiv.org/abs/2310.06825. Alex Kwon. Reclaim evaluation: A lossy memory is worse than an empty one,

    Pith/arXiv arXiv

  7. [8]

    Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg

    URL https://arxiv.org/abs/2606.25449. Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. InAdvances in Neural Information Processing Systems (NeurIPS),

    Pith/arXiv arXiv

  8. [9]

    Nils Reimers and Iryna Gurevych

    URLhttps://arxiv.org/abs/2412.15115. Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. InProceedings of EMNLP-IJCNLP,

    Pith/arXiv arXiv

  9. [11]

    Thomas Wolf et al

    URLhttps://arxiv.org/abs/2308.10248. Thomas Wolf et al. Transformers: State-of-the-art natural language processing. InProceedings of EMNLP: System Demonstrations, pp. 38–45,

    Pith/arXiv arXiv

  10. [13]

    Okay, here it is: the<object>. It works now

    URL https: //arxiv.org/abs/2310.01405. 11 APPENDICES A Stimuli and phrasings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12 B Probe and controls. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12 C Steer...

    Pith/arXiv arXiv

  11. [14]

    just got back from my first 5k, didn’t stop once

    A seventh model, Qwen2.5-32B, is run probe-and-honoring only: it represents the intent (probe 0.99 stated, 0.93inferred) and honors it at baseline (0.78/0.77), so it has no discard to steer (Section 7). L NATURALISTIC TRANSFER(FULL NUMBERS) A reader may suspect the synthetic, surface-matched stimuli drive the effect. We test whether the probe and the stee...

    2019