RECORD
AlgoTrace: Algorithmic Primitives and Compositional Geometry of Reasoning in Language Models
Machine Learning · Submitted 13 Oct 2025
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim These findings demonstrate that LLM reasoning can be understood as a walk through algorithmic primitives in the latent space governed by compositional geometry.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
How do inference time and latent computations enable large language models (LLMs) to solve multi-step reasoning problems? We introduce AlgoTrace, a framework for tracing and steering algorithmic operations in the model latent space for multi-step reasoning. We operationalize primitives by clustering latent activations of the model when solving four benchmarks: Traveling Salesperson Problem (TSP), 3SAT, AIME, and Graph Navigation. We annotate the clusters using their corresponding tokens in the reasoning trace. We then apply function vector methods to extract primitive vectors as reusable compositional building blocks of reasoning. We find that a) injecting a primitive vector into models (Phi, Qwen, Llama) elicits the associated algorithmic operation in the reasoning trace, b) injecting primitives can steer behavior across tasks, c) primitive vectors can be composed through algebraic operations, revealing a geometric logic in activation space, and d) a fine-tuned model exhibits improved composition of primitives (Phi-4-Reasoning vs. Phi-4). These findings demonstrate that LLM reasoning can be understood as a walk through algorithmic primitives in the latent space governed by compositional geometry. These primitives transfer across tasks, and reasoning finetuning strengthens algorithmic generalization and composition across domains.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | How do inference time and latent computations enable large language models (LLMs) to solve multi-step reasoning problems? | background 0.76 | 0.04 | 0.02 | unclear 1.00 |
| s1 | We introduce AlgoTrace, a framework for tracing and steering algorithmic operations in the model latent space for multi-step reasoning. | contribution 0.99 | 0.93 | 0.05 | asserted 0.98 |
| s2 | We operationalize primitives by clustering latent activations of the model when solving four benchmarks: Traveling Salesperson Problem (TSP), 3SAT, AIME, and Graph Navigation. | method 0.98 | 0.96 | 0.03 | asserted 0.97 |
| s3 | We annotate the clusters using their corresponding tokens in the reasoning trace. | method 1.00 | 0.95 | 0.02 | asserted 0.95 |
| s4 | We then apply function vector methods to extract primitive vectors as reusable compositional building blocks of reasoning. | method 0.99 | 0.95 | 0.03 | asserted 0.91 |
| s5 | We find that a) injecting a primitive vector into models (Phi, Qwen, Llama) elicits the associated algorithmic operation in the reasoning trace, b) injecting primitives can steer behavior across tasks, c) primitive vectors can be composed through algebraic operations, revealing a geometric logic in activation space, and d) a fine-tuned model exhibits improved composition of primitives (Phi-4-Reasoning vs. Phi-4). | result 1.00 | 0.96 | 0.29 | asserted 1.00 |
| s6 | These findings demonstrate that LLM reasoning can be understood as a walk through algorithmic primitives in the latent space governed by compositional geometry. | interpretation 1.00 | 0.74 | 0.86 | asserted 1.00 |
| s7 | These primitives transfer across tasks, and reasoning finetuning strengthens algorithmic generalization and composition across domains. | interpretation 0.53 | 0.85 | 0.17 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v3
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
How do inference time and latent computations enable large language models (LLMs) to solve multi-step reasoning problems? We introduce AlgoTrace, a framework for tracing and steering algorithmic operations in the model latent space for multi-step reasoning. We operationalize primitives by clustering latent activations of the model when solving four benchmarks: Traveling Salesperson Problem (TSP), 3SAT, AIME, and Graph Navigation. We annotate the clusters using their corresponding tokens in the reasoning trace. We then apply function vector methods to extract primitive vectors as reusable compositional building blocks of reasoning. We find that a) injecting a primitive vector into models (Phi, Qwen, Llama) elicits the associated algorithmic operation in the reasoning trace, b) injecting primitives can steer behavior across tasks, c) primitive vectors can be composed through algebraic operations, revealing a geometric logic in activation space, and d) a fine-tuned model exhibits improved composition of primitives (Phi-4-Reasoning vs. Phi-4). These findings demonstrate that LLM reasoning can be understood as a walk through algorithmic primitives in the latent space governed by compositional geometry. These primitives transfer across tasks, and reasoning finetuning strengthens algorithmic generalization and composition across domains.
Human review
References
How the literature uses it
5 arXiv papers cite this in 5 sentences
Papers citing it · 3 sentences without a confident label
Cites it as background 2 papers
-
Algorithmic Primitives extracts reusable primitive vectors from reasoning traces and activations and studies composition by addition, subtraction, and scaling [ref] .
A Pre-Specified Construction-Confirmation Test of Operation-Level Causal Transfer Across Finite Isomorphic Symbolic Domains · Successor mechanisms and algorithmic primitives · 0.94
-
Vector arithmetic in representation space Previous work has investigated the compositional nature of interventions [ref] , finding that steering vectors identified during ICL can be additive, allowing for the activation of task combinations through vector summation.
Distributed Sparse Interventions in Language Models · Related work · 0.92
What it cites · 20 sentences without a confident label
Cites as background 5 papers
-
Our work contributes to recent discussions about the extent to which LLMs exhibit abstract reasoning rather than relying on memorized procedures, failing to generalize in counterfactual scenarios [ref] .
Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets · Discussion · 0.88
-
Combined analysis of individual neurons [ref] , attention scores [ref] , and task-specific attention heads [ref] have further deepened our understanding of internal model processing.
Analyzing the Structure of Attention in a Transformer Language Model · Related Work · 0.92
-
Combined analysis of individual neurons [ref] , attention scores [ref] , and task-specific attention heads [ref] have further deepened our understanding of internal model processing.
Language Models Represent Space and Time · Related Work · 0.92
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.