Pith. sign in

REVIEW 3 major objections 6 minor 33 references

InTraVisTo: Inside Transformer Visualisation Tool

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read InTraVisTo lets users watch an LLM's internal token-by-token computation and edit it in real time without code.

desk verdict A genuinely useful open-source GUI tool whose headline Sankey 'information flow' is an unvalidated norm heuristic—useful as a tool paper, not as causal analysis. read the letter →

arxiv 2507.13858 v1 pith:VIOF65CB submitted 2025-07-18 cs.CL

classification cs.CL
keywords largelanguagemodelsTransformerexplainabilityhiddenstatedecodinginformationflowvisualizationembeddinginjectionSankeydiagraminterpretabilitytools
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper presents InTraVisTo, an interactive tool that opens up the inside of a Transformer-based large language model as it generates text. The tool decodes the model's hidden-state vectors at every layer into readable tokens, draws a Sankey diagram of how information accumulates toward the next-token prediction, and lets the user replace a hidden vector with a chosen token embedding to probe or correct the model. The central claim is that these features, combined in one interface and usable without programming, let researchers trace where a wrong answer goes wrong, for example, spotting that a feed-forward block at layer 29 contributes only 1.7% of the flow and that injecting an '8' there repairs a digit-reversal error. If true, this gives NLP practitioners a practical debugging and hypothesis-testing workflow for understanding LLM reasoning and failures.

What carries the argument

The carrying mechanism is a set of algebraic bridges between hidden states and vocabulary probabilities. Hidden states are decoded by RMS-normalizing the vector, multiplying by a decoder matrix, and applying softmax (Equation 1), with the decoder chosen as a depth-weighted linear interpolation between the input embedding and the output decoder (Equation 2). Information flow is quantified by the norm ratios in Equations 3 and 4, which assign each attention or feed-forward update a share of the residual norm, and recursively apportioned downward through Equation 8 to draw the Sankey diagram. Injection is handled by Equation 9, which removes the most probable token direction from a hidden state and replaces it with another token embedding scaled by the correlation between the two. Together these identities turn invisible vectors into token-level stories that a user can edit.

What would settle it

Run the same digit-reversal prompt on a model with the tool while performing activation patching on the layer-29 feed-forward block: replace the block's output at that position with its value from a correct run. If the erroneous '3' does not turn into '8', then the 1.7% flow attribution does not identify the causal source of the error, and the paper's central diagnostic claim is not supported.

Watch

Extended reading notes

Core claim

The paper claims that a single interactive tool can make the internal computation of a causal Transformer legible and editable at three levels: the decoded hidden states (which token the model 'thinks' at each layer and position), the relative information flow through attention, feed-forward, and residual components, and the effect of surgically injecting a token embedding into a chosen hidden state. The authors demonstrate the workflow on a number-reversal example in which the model outputs 43234381 instead of 43234831: the heatmap shows an '8' emerging in the feed-forward stream around layer 29, the flow attribution marks that feed-forward block as below-average (1.7% versus at least 1.8% for other contributions at that layer), and injecting the embedding of '8' at that position changes the output to the correct reversal. On those grounds the paper argues that the tool can localize a reasoning error to a specific component and layer and test the hypothesis interactively.

Load-bearing premise

The load-bearing premise is that the norm-based attribution ratios in Equations 3-8 measure the actual causal information flow from lower to higher layers, even though the paper explicitly chooses this as a minimal heuristic rather than validating it against path patching or other causal ablations.

Editorial extensions

If this is right

  • A user with no programming background can examine any causal Transformer whose hidden states are accessible and follow how a specific token prediction is built layer by layer.
  • Diagnosing a wrong output becomes a concrete search: compare heatmap layers for a low-probability step, read the relative flow percentages, and form a hypothesis about which component failed to propagate the right information.
  • The embedding injection gives a direct test of such hypotheses: replacing a token direction at the suspected layer either fixes the output (supporting the diagnosis) or leaves it unchanged (pointing elsewhere).
  • The tool combines three previously separate capabilities, hidden-state decoding, flow visualization, and intervention, in one interface, which is the paper's stated novelty over existing tools.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable extension the paper does not perform is comparing its 1.7% flow figure for the layer-29 feed-forward block against activation patching on the same example; without that comparison, the Sankey percentages should be treated as heuristic attributions rather than measured causes.
  • The depth-interpolated decoder in Equation 2 could serve as a training-free baseline for hidden-state interpretability on any model with untied input and output embeddings, a use the paper does not itself claim.
  • If the late-layer drop in decoded probability that the paper observes in the erroneous digit generalizes across tasks, the same heatmap could be used as an online confidence or error detector during text generation, a direction the authors do not develop.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents InTraVisTo, an interactive visualization tool for transformer LLMs that combines three capabilities: decoding internal hidden states into token probabilities via an interpolated decoder (Eq. 2), visualizing information flow through a Sankey diagram using a norm-based attribution scheme (Eqs. 3-8), and probing the model by injecting embeddings into hidden states (Eq. 9). The tool is demonstrated on a digit-reversal task with Mistral-7B-Instruct, where a wrong output is attributed to a low-contribution feed-forward block at layer 29. The paper claims the tool enables researchers and non-experts to interpret and manipulate LLMs in real time without programming expertise.

Significance. The open-source implementation and live demo are concrete strengths, and the combination of hidden-state decoding, flow visualization, and injection in a single GUI is, to my knowledge, novel among publicly available tools. The attribution equations are explicit, which makes the tool's behavior reproducible. However, the scientific contribution of the diagnostic narrative is limited by the lack of validation of the norm-based flow measure against causal interventions. If the flow metric were shown to track causal influence, the tool would be a valuable addition to mechanistic interpretability; as it stands, it is a plausible and useful exploratory visualization whose interpretive claims need support.

major comments (3)
  1. [II-C, Eqs. (3)-(8)] The information-flow percentages are computed from vector norms and attention weights, not from causal intervention. In a residual stream, a small-norm update can have decisive effect if aligned with the decoding direction, and attention weights do not measure value contribution. The paper uses this metric to conclude that the FF block at layer 29 has low contribution (Section II-C, Fig. 7) and that the model 'forgets the final target' (Section II-D). The manuscript concedes the heuristic nature in Section III-B and defers causal analysis to future work in the Conclusion. As a result, the central diagnostic claim is unsupported. A validation on at least the running example—e.g., path patching the layer-29 FF output—is required, or the claims should be reframed as descriptive heuristics.
  2. [II-B, Fig. 5] The case study infers, from manual inspection of heatmaps, that an 'incorrect or imprecise internal representation around the 29th layer' caused the error. This is a single example with no quantitative support or comparison to correct generations. The paper should either label this as an anecdotal hypothesis or back it with a controlled analysis (e.g., comparing many correct and incorrect reversals).
  3. [II-D, Eq. (9)] The injection formula is described as scaling by the 'correlation' between eold and h, but h·eold is a dot product, not a correlation unless centering is defined. More substantively, the claim that injecting an 8 'solves' the error is undermined by the paper's own observation that the output gains an extra digit; the conclusion should be revised to reflect the mixed result.
minor comments (6)
  1. [Throughout] Several typographical errors appear: 'hypotesise' (Section II-B), 'sates' (Section III-A), 'InTraVisToto' (Conclusion), and 'it's prediction' (Section II-B) should be 'its prediction'.
  2. [II-C, Eq. (8)] The summation index and the scope of the residual term should be clarified; define [j,k] and indicate that the (1−%att) term is the bypass flow.
  3. [II-D, Eq. (9)] The notation h·eold is used for both the dot product and the product h·eold·(enew−eold); consider using a different symbol for the dot product and explain the scaling factor.
  4. [II-B, Eq. (2)] The interpolation is linear in layer index; it may be worth noting that for models with tied embeddings the decoder is exact, and that the chosen decoder affects the interpretation of heatmap cells.
  5. [Figures 5 and 7] The captions of Figures 5 and 7 repeat the same sentence verbatim; consider differentiating them to avoid confusion.
  6. [II-C] The paper uses the term 'contribution' for the Sankey flow, but Eqs. (3)-(4) define a norm-based share; the text should consistently call this a 'norm-based share' or 'flow share' to avoid over-interpretation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the tool's flow attribution is a stipulated heuristic, explicitly non-causal, and no result is fitted to or defined by itself.

full rationale

InTraVisTo is a visualization tool, not a derivation of predictions from fitted parameters. The Sankey information-flow percentages are computed from Equations 3-8, which define the metric; observing that the layer-29 FF block carries 1.7% of flow is a direct calculation from the tool's own stipulated definition, not a prediction validated against external data. The paper nowhere claims this norm-based flow equals causal effect: Section III-B explicitly states 'we resort to a more minimal flow interpretation due to the high computational cost of activation and path patching experiments,' and the Conclusion lists causal analysis via mechanistic interpretability as future work. No parameter is fitted to a subset and then 'predicted'; no load-bearing argument relies on a self-citation (the interpretability works cited, such as Logit Lens, Tuned Lens, and Circuit Transformers, are all external, with no author overlap); and the injection experiment is an independent intervention that tests, rather than defines, the hypothesis formed from the heatmap. The gap between norm-based attribution and true causal contribution is a validation limitation, not a circularity, because the paper does not convert its own metric's output into an externally forced conclusion. Therefore score 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the interpretability assumptions inherited from the logit lens literature, plus a new norm-based attribution heuristic that the paper does not validate. No numeric parameters are fitted to data, and no new physical or conceptual entities are introduced.

assumptions (4)
  • domain assumption Decoding hidden states via the model's own embeddings (Eq. 1) yields meaningful token interpretations.
    Basis of the entire heatmap visualization; inherited from Logit Lens [15] and not verified in this paper.
  • ad hoc to paper The vector-norm ratios (Eqs. 3-4) capture the relative contribution of attention and feed-forward components to the residual stream.
    This is a heuristic introduced by the authors; no causal validation is provided.
  • domain assumption Injecting a token embedding (Eq. 9) is a valid causal intervention that reveals internal processing without confounding.
    Activation patching assumes this, but the paper does not test that the intervention is clean.
  • domain assumption The residual stream is approximately linear so that adding a scaled embedding (Eq. 9) cleanly replaces the old component.
    The paper relies on orthogonality/decomposition properties of the residual stream that are not established here.

how reviews work

0 comments
Cite this review

Pith. "Pith review of InTraVisTo: Inside Transformer Visualisation Tool." pith.science (2026). https://pith.science/paper/VIOF65CB

@misc{pith2026250713858,
  author       = {Pith},
  title        = {Pith review of: InTraVisTo: Inside Transformer Visualisation Tool},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VIOF65CB}},
  note         = {Machine review of arXiv:2507.13858}
}
read the original abstract

The reasoning capabilities of Large Language Models (LLMs) have increased greatly over the last few years, as have their size and complexity. Nonetheless, the use of LLMs in production remains challenging due to their unpredictable nature and discrepancies that can exist between their desired behavior and their actual model output. In this paper, we introduce a new tool, InTraVisTo (Inside Transformer Visualisation Tool), designed to enable researchers to investigate and trace the computational process that generates each token in a Transformer-based LLM. InTraVisTo provides a visualization of both the internal state of the Transformer model (by decoding token embeddings at each layer of the model) and the information flow between the various components across the different layers of the model (using a Sankey diagram). With InTraVisTo, we aim to help researchers and practitioners better understand the computations being performed within the Transformer model and thus to shed some light on internal patterns and reasoning processes employed by LLMs.

Figures

Figures reproduced from arXiv: 2507.13858 by the authors.

Figure 1
Figure 1. InTraVisTo application overview: given an input [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The main control panel of InTraVisTo. The [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Transformer model architecture. The high-level view of the architecture is represented on the left. In the centre there [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Hidden states heatmap generated by InTraVisTo using the configuration depicted in Figure 2. The input prompt was [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Slice of a heatmap from erroneous generation with [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Top 5 layers of a Sankey diagram generated by InTraVisTo using the configuration depicted in Figure 2. The input [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Flow percentages visualised by hovering the mouse [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 16 canonical work pages

  1. [1]

    Evaluating large language models as agents in the clinic,

    N. Mehandru, B. Y . Miao, E. R. Almaraz, M. Sushil, A. J. Butte, and A. M. Alaa, “Evaluating large language models as agents in the clinic,” npj Digit. Medicine , vol. 7, no. 1, 2024. [Online]. Available: https://doi.org/10.1038/s41746-024-01083-y

  2. [2]

    Bloomberggpt: A large language model for finance,

    S. Wu, O. Irsoy, S. Lu, V . Dabravolski, M. Dredze, S. Gehrmann et al. , “Bloomberggpt: A large language model for finance,” CoRR, vol. abs/2303.17564, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2303.17564

  3. [3]

    Training compute-optimal large language models,

    J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford et al. , “Training compute-optimal large language models,” CoRR, vol. abs/2203.15556, 2022. [Online]. Available: https://doi.org/10.48550/arXiv.2203.15556

  4. [4]

    GPT-4 technical report,

    OpenAI, “GPT-4 technical report,” CoRR, vol. abs/2303.08774, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2303.08774

  5. [5]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez et al. , “Attention is all you need,” in Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V . N. Vishwanatha...

  6. [6]

    Survey of hallucination in natural language generation,

    Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y . Xu et al. , “Survey of hallucination in natural language generation,” ACM Comput. Surv., vol. 55, no. 12, pp. 248:1–248:38, 2023. [Online]. Available: https://doi.org/10.1145/3571730

  7. [7]

    Exposing attention glitches with flip-flop language modeling,

    B. Liu, J. T. Ash, S. Goel, A. Krishnamurthy, and C. Zhang, “Exposing attention glitches with flip-flop language modeling,” in Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Har...

  8. [8]

    Formal algorithms for transformers,

    M. Phuong and M. Hutter, “Formal algorithms for transformers,” CoRR, vol. abs/2207.09238, 2022. [Online]. Available: https://doi.org/10.48550/arXiv.2207.09238

Show all 33 references
  1. [9]

    Lan- guage models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Lan- guage models are unsupervised multitask learners,” OpenAI Blog, 2019

  2. [10]

    Gemma: Open models based on gemini research and technology,

    T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre et al., “Gemma: Open models based on gemini research and technology,” CoRR, vol. abs/2403.08295, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2403.08295

  3. [11]

    Mistral 7b,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de Las Casas et al., “Mistral 7b,” CoRR, vol. abs/2310.06825, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2310.06825

  4. [12]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei et al. , “Llama 2: Open foundation and fine-tuned chat models,” CoRR, vol. abs/2307.09288, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2307.09288

  5. [13]

    Eliciting latent predictions from transformers with the tuned lens,

    N. Belrose, Z. Furman, L. Smith, D. Halawi, I. Ostrovsky, L. McKinney et al. , “Eliciting latent predictions from transformers with the tuned lens,” CoRR, vol. abs/2303.08112, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2303.08112

  6. [14]

    A primer on the inner workings of transformer-based language models,

    J. Ferrando, G. Sarti, A. Bisazza, and M. R. Costa-juss `a, “A primer on the inner workings of transformer-based language models,” CoRR, vol. abs/2405.00208, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2405.00208

  7. [15]

    (2020) Interpreting gpt: the logit lens

    nostalgebraist. (2020) Interpreting gpt: the logit lens. [Online]. Available: https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting -gpt-the-logit-lens

  8. [16]

    Attention lens: A tool for mechanistically interpreting the attention head information retrieval mechanism,

    M. Sakarvadia, A. Khan, A. Ajith, D. Grzenda, N. Hudson, A. Bauer et al., “Attention lens: A tool for mechanistically interpreting the attention head information retrieval mechanism,” CoRR, vol. abs/2310.16270,

  9. [17]

    Future lens: An- ticipating subsequent tokens from a single hidden state,

    K. Pal, J. Sun, A. Yuan, B. C. Wallace, and D. Bau, “Future lens: An- ticipating subsequent tokens from a single hidden state,” in Proceedings of the 27th Conference on Computational Natural Language Learning, CoNLL 2023, Singapore, December 6-7, 2023 , J. Jiang, D. Reitter, a...

  10. [18]

    Understanding intermediate layers using linear classifier probes,

    G. Alain and Y . Bengio, “Understanding intermediate layers using linear classifier probes,” in 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings . OpenReview.net, 2017. [Online]. Available: https...

  11. [19]

    A multiscale visualization of attention in the transformer model,

    J. Vig, “A multiscale visualization of attention in the transformer model,” in Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28 - August 2, 2019, Volume 3: System Demonstrations , M. R. Costa-juss `a and E....

  12. [20]

    Dodrio: Exploring transformer models with interactive visualization,

    Z. J. Wang, R. Turko, and D. H. Chau, “Dodrio: Exploring transformer models with interactive visualization,” in Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natura...

  13. [21]

    LM transparency tool: Interactive tool for analyzing transformer language models,

    I. Tufanov, K. Hambardzumyan, J. Ferrando, and E. V oita, “LM transparency tool: Interactive tool for analyzing transformer language models,” CoRR, vol. abs/2404.07004, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2404.07004

  14. [22]

    Elhage, N

    N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann et al. (2021) A mathematical framework for transformer circuits. [Online]. Available: https://transformer-circuits.pub/2021/framework/index.html

  15. [23]

    Interpretability in the wild: a circuit for indirect object identification in GPT-2 small,

    K. R. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt, “Interpretability in the wild: a circuit for indirect object identification in GPT-2 small,” in The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . O...

  16. [24]

    ”why should I trust you?

    M. T. Ribeiro, S. Singh, and C. Guestrin, “”why should I trust you?”: Explaining the predictions of any classifier,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016 , B. Krishnapu...

  17. [25]

    Causal abstractions of neural networks,

    A. Geiger, H. Lu, T. Icard, and C. Potts, “Causal abstractions of neural networks,” in Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , M. Ranzato, A. Beygelzimer...

  18. [26]

    Patchscopes: A unifying framework for inspecting hidden representations of language models,

    A. Ghandeharioun, A. Caciularu, A. Pearce, L. Dixon, and M. Geva, “Patchscopes: A unifying framework for inspecting hidden representations of language models,” CoRR, vol. abs/2401.06102, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2401.06102

  19. [27]

    Elhage, N

    N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann et al. (2021) Transformerlens. [Online]. Available: https://transformer-circuits.pub/2021/garcon/index.html

  20. [28]

    Transformerlens,

    N. Nanda and J. Bloom, “Transformerlens,” https://github.com/Trans formerLensOrg/TransformerLens, 2022

  21. [29]

    How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model,

    M. Hanna, O. Liu, and A. Variengien, “How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model,” in Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2...

  22. [30]

    Towards automated circuit discovery for mechanistic interpretability,

    A. Conmy, A. N. Mavor-Parker, A. Lynch, S. Heimersheim, and A. Garriga-Alonso, “Towards automated circuit discovery for mechanistic interpretability,” in Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, Neur...

  23. [31]

    Available: http://papers.nips.cc/paper \ files/paper/202 3/hash/efbba7719cc5172d175240f24be11280-Abstract-Conference.html

    [Online]. Available: http://papers.nips.cc/paper \ files/paper/202 3/hash/efbba7719cc5172d175240f24be11280-Abstract-Conference.html

  24. [33]

    Available: http://papers.nips.cc/paper \ files/paper/202 3/hash/34e1dbe95d34d7ebaf99b9bcaeb5b2be-Abstract-Conference.html

    [Online]. Available: http://papers.nips.cc/paper \ files/paper/202 3/hash/34e1dbe95d34d7ebaf99b9bcaeb5b2be-Abstract-Conference.html

  25. [2023]

    Available: https://doi.org/10.48550/arXiv.2310.16270

    [Online]. Available: https://doi.org/10.48550/arXiv.2310.16270

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.