REVIEW 3 major objections 6 minor 33 references
InTraVisTo: Inside Transformer Visualisation Tool
T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read InTraVisTo lets users watch an LLM's internal token-by-token computation and edit it in real time without code.
desk verdict A genuinely useful open-source GUI tool whose headline Sankey 'information flow' is an unvalidated norm heuristic—useful as a tool paper, not as causal analysis. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is a set of algebraic bridges between hidden states and vocabulary probabilities. Hidden states are decoded by RMS-normalizing the vector, multiplying by a decoder matrix, and applying softmax (Equation 1), with the decoder chosen as a depth-weighted linear interpolation between the input embedding and the output decoder (Equation 2). Information flow is quantified by the norm ratios in Equations 3 and 4, which assign each attention or feed-forward update a share of the residual norm, and recursively apportioned downward through Equation 8 to draw the Sankey diagram. Injection is handled by Equation 9, which removes the most probable token direction from a hidden state and replaces it with another token embedding scaled by the correlation between the two. Together these identities turn invisible vectors into token-level stories that a user can edit.
What would settle it
Run the same digit-reversal prompt on a model with the tool while performing activation patching on the layer-29 feed-forward block: replace the block's output at that position with its value from a correct run. If the erroneous '3' does not turn into '8', then the 1.7% flow attribution does not identify the causal source of the error, and the paper's central diagnostic claim is not supported.
Extended reading notes
Core claim
The paper claims that a single interactive tool can make the internal computation of a causal Transformer legible and editable at three levels: the decoded hidden states (which token the model 'thinks' at each layer and position), the relative information flow through attention, feed-forward, and residual components, and the effect of surgically injecting a token embedding into a chosen hidden state. The authors demonstrate the workflow on a number-reversal example in which the model outputs 43234381 instead of 43234831: the heatmap shows an '8' emerging in the feed-forward stream around layer 29, the flow attribution marks that feed-forward block as below-average (1.7% versus at least 1.8% for other contributions at that layer), and injecting the embedding of '8' at that position changes the output to the correct reversal. On those grounds the paper argues that the tool can localize a reasoning error to a specific component and layer and test the hypothesis interactively.
Load-bearing premise
The load-bearing premise is that the norm-based attribution ratios in Equations 3-8 measure the actual causal information flow from lower to higher layers, even though the paper explicitly chooses this as a minimal heuristic rather than validating it against path patching or other causal ablations.
Editorial extensions
If this is right
- A user with no programming background can examine any causal Transformer whose hidden states are accessible and follow how a specific token prediction is built layer by layer.
- Diagnosing a wrong output becomes a concrete search: compare heatmap layers for a low-probability step, read the relative flow percentages, and form a hypothesis about which component failed to propagate the right information.
- The embedding injection gives a direct test of such hypotheses: replacing a token direction at the suspected layer either fixes the output (supporting the diagnosis) or leaves it unchanged (pointing elsewhere).
- The tool combines three previously separate capabilities, hidden-state decoding, flow visualization, and intervention, in one interface, which is the paper's stated novelty over existing tools.
Reading between the lines
- One testable extension the paper does not perform is comparing its 1.7% flow figure for the layer-29 feed-forward block against activation patching on the same example; without that comparison, the Sankey percentages should be treated as heuristic attributions rather than measured causes.
- The depth-interpolated decoder in Equation 2 could serve as a training-free baseline for hidden-state interpretability on any model with untied input and output embeddings, a use the paper does not itself claim.
- If the late-layer drop in decoded probability that the paper observes in the erroneous digit generalizes across tasks, the same heatmap could be used as an online confidence or error detector during text generation, a direction the authors do not develop.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents InTraVisTo, an interactive visualization tool for transformer LLMs that combines three capabilities: decoding internal hidden states into token probabilities via an interpolated decoder (Eq. 2), visualizing information flow through a Sankey diagram using a norm-based attribution scheme (Eqs. 3-8), and probing the model by injecting embeddings into hidden states (Eq. 9). The tool is demonstrated on a digit-reversal task with Mistral-7B-Instruct, where a wrong output is attributed to a low-contribution feed-forward block at layer 29. The paper claims the tool enables researchers and non-experts to interpret and manipulate LLMs in real time without programming expertise.
Significance. The open-source implementation and live demo are concrete strengths, and the combination of hidden-state decoding, flow visualization, and injection in a single GUI is, to my knowledge, novel among publicly available tools. The attribution equations are explicit, which makes the tool's behavior reproducible. However, the scientific contribution of the diagnostic narrative is limited by the lack of validation of the norm-based flow measure against causal interventions. If the flow metric were shown to track causal influence, the tool would be a valuable addition to mechanistic interpretability; as it stands, it is a plausible and useful exploratory visualization whose interpretive claims need support.
major comments (3)
- [II-C, Eqs. (3)-(8)] The information-flow percentages are computed from vector norms and attention weights, not from causal intervention. In a residual stream, a small-norm update can have decisive effect if aligned with the decoding direction, and attention weights do not measure value contribution. The paper uses this metric to conclude that the FF block at layer 29 has low contribution (Section II-C, Fig. 7) and that the model 'forgets the final target' (Section II-D). The manuscript concedes the heuristic nature in Section III-B and defers causal analysis to future work in the Conclusion. As a result, the central diagnostic claim is unsupported. A validation on at least the running example—e.g., path patching the layer-29 FF output—is required, or the claims should be reframed as descriptive heuristics.
- [II-B, Fig. 5] The case study infers, from manual inspection of heatmaps, that an 'incorrect or imprecise internal representation around the 29th layer' caused the error. This is a single example with no quantitative support or comparison to correct generations. The paper should either label this as an anecdotal hypothesis or back it with a controlled analysis (e.g., comparing many correct and incorrect reversals).
- [II-D, Eq. (9)] The injection formula is described as scaling by the 'correlation' between eold and h, but h·eold is a dot product, not a correlation unless centering is defined. More substantively, the claim that injecting an 8 'solves' the error is undermined by the paper's own observation that the output gains an extra digit; the conclusion should be revised to reflect the mixed result.
minor comments (6)
- [Throughout] Several typographical errors appear: 'hypotesise' (Section II-B), 'sates' (Section III-A), 'InTraVisToto' (Conclusion), and 'it's prediction' (Section II-B) should be 'its prediction'.
- [II-C, Eq. (8)] The summation index and the scope of the residual term should be clarified; define [j,k] and indicate that the (1−%att) term is the bypass flow.
- [II-D, Eq. (9)] The notation h·eold is used for both the dot product and the product h·eold·(enew−eold); consider using a different symbol for the dot product and explain the scaling factor.
- [II-B, Eq. (2)] The interpolation is linear in layer index; it may be worth noting that for models with tied embeddings the decoder is exact, and that the chosen decoder affects the interpretation of heatmap cells.
- [Figures 5 and 7] The captions of Figures 5 and 7 repeat the same sentence verbatim; consider differentiating them to avoid confusion.
- [II-C] The paper uses the term 'contribution' for the Sankey flow, but Eqs. (3)-(4) define a norm-based share; the text should consistently call this a 'norm-based share' or 'flow share' to avoid over-interpretation.
Circularity Check
No circular derivation: the tool's flow attribution is a stipulated heuristic, explicitly non-causal, and no result is fitted to or defined by itself.
full rationale
InTraVisTo is a visualization tool, not a derivation of predictions from fitted parameters. The Sankey information-flow percentages are computed from Equations 3-8, which define the metric; observing that the layer-29 FF block carries 1.7% of flow is a direct calculation from the tool's own stipulated definition, not a prediction validated against external data. The paper nowhere claims this norm-based flow equals causal effect: Section III-B explicitly states 'we resort to a more minimal flow interpretation due to the high computational cost of activation and path patching experiments,' and the Conclusion lists causal analysis via mechanistic interpretability as future work. No parameter is fitted to a subset and then 'predicted'; no load-bearing argument relies on a self-citation (the interpretability works cited, such as Logit Lens, Tuned Lens, and Circuit Transformers, are all external, with no author overlap); and the injection experiment is an independent intervention that tests, rather than defines, the hypothesis formed from the heatmap. The gap between norm-based attribution and true causal contribution is a validation limitation, not a circularity, because the paper does not convert its own metric's output into an externally forced conclusion. Therefore score 0.
Assumptions & free parameters
assumptions (4)
- domain assumption Decoding hidden states via the model's own embeddings (Eq. 1) yields meaningful token interpretations.
- ad hoc to paper The vector-norm ratios (Eqs. 3-4) capture the relative contribution of attention and feed-forward components to the residual stream.
- domain assumption Injecting a token embedding (Eq. 9) is a valid causal intervention that reveals internal processing without confounding.
- domain assumption The residual stream is approximately linear so that adding a scaled embedding (Eq. 9) cleanly replaces the old component.
Cite this review
Pith. "Pith review of InTraVisTo: Inside Transformer Visualisation Tool." pith.science (2026). https://pith.science/paper/VIOF65CB
@misc{pith2026250713858,
author = {Pith},
title = {Pith review of: InTraVisTo: Inside Transformer Visualisation Tool},
year = {2026},
howpublished = {\url{https://pith.science/paper/VIOF65CB}},
note = {Machine review of arXiv:2507.13858}
}
read the original abstract
The reasoning capabilities of Large Language Models (LLMs) have increased greatly over the last few years, as have their size and complexity. Nonetheless, the use of LLMs in production remains challenging due to their unpredictable nature and discrepancies that can exist between their desired behavior and their actual model output. In this paper, we introduce a new tool, InTraVisTo (Inside Transformer Visualisation Tool), designed to enable researchers to investigate and trace the computational process that generates each token in a Transformer-based LLM. InTraVisTo provides a visualization of both the internal state of the Transformer model (by decoding token embeddings at each layer of the model) and the information flow between the various components across the different layers of the model (using a Sankey diagram). With InTraVisTo, we aim to help researchers and practitioners better understand the computations being performed within the Transformer model and thus to shed some light on internal patterns and reasoning processes employed by LLMs.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Evaluating large language models as agents in the clinic,
N. Mehandru, B. Y . Miao, E. R. Almaraz, M. Sushil, A. J. Butte, and A. M. Alaa, “Evaluating large language models as agents in the clinic,” npj Digit. Medicine , vol. 7, no. 1, 2024. [Online]. Available: https://doi.org/10.1038/s41746-024-01083-y
-
[2]
Bloomberggpt: A large language model for finance,
S. Wu, O. Irsoy, S. Lu, V . Dabravolski, M. Dredze, S. Gehrmann et al. , “Bloomberggpt: A large language model for finance,” CoRR, vol. abs/2303.17564, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2303.17564
-
[3]
Training compute-optimal large language models,
J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford et al. , “Training compute-optimal large language models,” CoRR, vol. abs/2203.15556, 2022. [Online]. Available: https://doi.org/10.48550/arXiv.2203.15556
-
[4]
OpenAI, “GPT-4 technical report,” CoRR, vol. abs/2303.08774, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2303.08774
-
[5]
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez et al. , “Attention is all you need,” in Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V . N. Vishwanatha...
work page 2017
-
[6]
Survey of hallucination in natural language generation,
Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y . Xu et al. , “Survey of hallucination in natural language generation,” ACM Comput. Surv., vol. 55, no. 12, pp. 248:1–248:38, 2023. [Online]. Available: https://doi.org/10.1145/3571730
doi:10.1145/3571730 2023
-
[7]
Exposing attention glitches with flip-flop language modeling,
B. Liu, J. T. Ash, S. Goel, A. Krishnamurthy, and C. Zhang, “Exposing attention glitches with flip-flop language modeling,” in Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Har...
work page 2023
-
[8]
Formal algorithms for transformers,
M. Phuong and M. Hutter, “Formal algorithms for transformers,” CoRR, vol. abs/2207.09238, 2022. [Online]. Available: https://doi.org/10.48550/arXiv.2207.09238
Show all 33 references
-
[9]
Lan- guage models are unsupervised multitask learners,
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Lan- guage models are unsupervised multitask learners,” OpenAI Blog, 2019
2019
-
[10]
Gemma: Open models based on gemini research and technology,
T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre et al., “Gemma: Open models based on gemini research and technology,” CoRR, vol. abs/2403.08295, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2403.08295
- [11]
- [12]
-
[13]
Eliciting latent predictions from transformers with the tuned lens,
N. Belrose, Z. Furman, L. Smith, D. Halawi, I. Ostrovsky, L. McKinney et al. , “Eliciting latent predictions from transformers with the tuned lens,” CoRR, vol. abs/2303.08112, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2303.08112
-
[14]
A primer on the inner workings of transformer-based language models,
J. Ferrando, G. Sarti, A. Bisazza, and M. R. Costa-juss `a, “A primer on the inner workings of transformer-based language models,” CoRR, vol. abs/2405.00208, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2405.00208
-
[15]
(2020) Interpreting gpt: the logit lens
nostalgebraist. (2020) Interpreting gpt: the logit lens. [Online]. Available: https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting -gpt-the-logit-lens
2020
-
[16]
Attention lens: A tool for mechanistically interpreting the attention head information retrieval mechanism,
M. Sakarvadia, A. Khan, A. Ajith, D. Grzenda, N. Hudson, A. Bauer et al., “Attention lens: A tool for mechanistically interpreting the attention head information retrieval mechanism,” CoRR, vol. abs/2310.16270,
-
[17]
Future lens: An- ticipating subsequent tokens from a single hidden state,
K. Pal, J. Sun, A. Yuan, B. C. Wallace, and D. Bau, “Future lens: An- ticipating subsequent tokens from a single hidden state,” in Proceedings of the 27th Conference on Computational Natural Language Learning, CoNLL 2023, Singapore, December 6-7, 2023 , J. Jiang, D. Reitter, a...
2023 doi
-
[18]
Understanding intermediate layers using linear classifier probes,
G. Alain and Y . Bengio, “Understanding intermediate layers using linear classifier probes,” in 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings . OpenReview.net, 2017. [Online]. Available: https...
2017
-
[19]
A multiscale visualization of attention in the transformer model,
J. Vig, “A multiscale visualization of attention in the transformer model,” in Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28 - August 2, 2019, Volume 3: System Demonstrations , M. R. Costa-juss `a and E....
2019 doi
-
[20]
Dodrio: Exploring transformer models with interactive visualization,
Z. J. Wang, R. Turko, and D. H. Chau, “Dodrio: Exploring transformer models with interactive visualization,” in Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natura...
2021
-
[21]
LM transparency tool: Interactive tool for analyzing transformer language models,
I. Tufanov, K. Hambardzumyan, J. Ferrando, and E. V oita, “LM transparency tool: Interactive tool for analyzing transformer language models,” CoRR, vol. abs/2404.07004, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2404.07004
-
[22]
Elhage, N
N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann et al. (2021) A mathematical framework for transformer circuits. [Online]. Available: https://transformer-circuits.pub/2021/framework/index.html
2021
-
[23]
Interpretability in the wild: a circuit for indirect object identification in GPT-2 small,
K. R. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt, “Interpretability in the wild: a circuit for indirect object identification in GPT-2 small,” in The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . O...
2023
-
[24]
”why should I trust you?
M. T. Ribeiro, S. Singh, and C. Guestrin, “”why should I trust you?”: Explaining the predictions of any classifier,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016 , B. Krishnapu...
2016
-
[25]
Causal abstractions of neural networks,
A. Geiger, H. Lu, T. Icard, and C. Potts, “Causal abstractions of neural networks,” in Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , M. Ranzato, A. Beygelzimer...
2021
-
[26]
Patchscopes: A unifying framework for inspecting hidden representations of language models,
A. Ghandeharioun, A. Caciularu, A. Pearce, L. Dixon, and M. Geva, “Patchscopes: A unifying framework for inspecting hidden representations of language models,” CoRR, vol. abs/2401.06102, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2401.06102
-
[27]
Elhage, N
N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann et al. (2021) Transformerlens. [Online]. Available: https://transformer-circuits.pub/2021/garcon/index.html
2021
-
[28]
Transformerlens,
N. Nanda and J. Bloom, “Transformerlens,” https://github.com/Trans formerLensOrg/TransformerLens, 2022
2022
-
[29]
How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model,
M. Hanna, O. Liu, and A. Variengien, “How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model,” in Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2...
2023
-
[30]
Towards automated circuit discovery for mechanistic interpretability,
A. Conmy, A. N. Mavor-Parker, A. Lynch, S. Heimersheim, and A. Garriga-Alonso, “Towards automated circuit discovery for mechanistic interpretability,” in Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, Neur...
2023
-
[31]
Available: http://papers.nips.cc/paper \ files/paper/202 3/hash/efbba7719cc5172d175240f24be11280-Abstract-Conference.html
[Online]. Available: http://papers.nips.cc/paper \ files/paper/202 3/hash/efbba7719cc5172d175240f24be11280-Abstract-Conference.html
-
[33]
Available: http://papers.nips.cc/paper \ files/paper/202 3/hash/34e1dbe95d34d7ebaf99b9bcaeb5b2be-Abstract-Conference.html
[Online]. Available: http://papers.nips.cc/paper \ files/paper/202 3/hash/34e1dbe95d34d7ebaf99b9bcaeb5b2be-Abstract-Conference.html
- [2023]
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.