Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Log-Augmented Generation: Scaling Test-Time Reasoning with Reusable Computation

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that storing and re-injecting the key-value states of past reasoning traces lets LLM agents solve new tasks more accurately and in fewer steps.

desk verdict New mechanism worth a referee's time, but the missing random-KV control means the reuse claim is under-determined. read the letter →

arxiv 2505.14398 v1 pith:45IA3SBG submitted 2025-05-20 cs.CL cs.AIcs.LG

Peter Baile Chen , Yi Zhang , Dan Roth , Samuel Madden , Jacob Andreas , Michael Cafarella This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords log-augmentedgenerationKVcachereusetest-timereasoningagenticretrieval-augmentedmulti-hopquestionansweringtracememoryretrievalaugmentationtransformerattentionstates
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that an LLM agent can learn from past experience by reusing the computation itself, not a summary of it. Its log-augmented generation (LAG) framework stores the key-value (KV) attention states of previous reasoning traces, retrieves the most relevant ones for a new question, and injects them into the model's context. In the LAGKV implementation the model encodes all earlier model responses in a task but stores only the KV values of the last response, relying on the attention mechanism's context-aggregation property to preserve the whole reasoning trace compactly. The paper reports that this approach outperforms standard agentic systems, reflection-based memory, and existing KV-cache methods on knowledge-intensive multi-hop QA and on reasoning-intensive math and science benchmarks, while also reducing the number of reasoning steps. If the mechanism holds, agents could accumulate reusable reasoning across tasks without extra distillation, fine-tuning, or retraining.

What carries the argument

The load-bearing object is the KV-cache log representation used by LAGKV: the key-value attention states of the tokens in the last model response, computed while the model attends to the entire multi-turn reasoning trace. This is what lets the method store a small token subset yet claim to preserve full context. The second piece of machinery is rotary-position re-embedding: stored KV caches are stripped of their original RoPE rotation and re-rotated with the positional IDs of the new context before being concatenated with the model's working context.

What would settle it

Run LAGKV again but replace the retrieved KV logs with KV caches taken from unrelated questions (or with random vectors of the same shape); if exact match stays near the reported LAGKV level, the gain is not coming from reused reasoning, whereas a drop back toward the no-log baseline would confirm the reuse mechanism is doing the work.

Watch

Extended reading notes

Core claim

The central claim, stated on the paper's own terms, is that directly reusing prior reasoning via KV caches is a better form of test-time memory than re-deriving, reflecting on, or distilling that reasoning into text. Concretely, LAGKV encodes the full reasoning context of a previous task into KV values, stores only the KV values of the tokens from the last model response, and, when a new task arrives, retrieves the most relevant of these stored caches by semantic similarity and feeds them into the model's context. Because each token's KV value is a weighted aggregation of all preceding context, the stored last-response caches are claimed to carry the essential reasoning of the whole trace. Across Musique, 2WikiMultiHop, GPQA, and MMLU-Pro, the paper reports that LAGKV beats a standard agentic system, a reflection-based memory method, and a KV-cache baseline on exact match while using fewer iterations, with the gains attributed to both knowledge reuse and insight reuse.

Load-bearing premise

The load-bearing premise is that a token's stored key-value state from an earlier reasoning trace keeps its meaning when moved into a new question's context; if attention states do not transfer that way, the reported accuracy gains disappear.

Editorial extensions

If this is right

  • Agent systems can drop explicit reflection or distillation steps, because directly injecting stored KV states from prior tasks yields better accuracy than first converting logs into abstract insights.
  • Since KV cache size depends only on the number of stored tokens and not on the amount of encoded context, a full reasoning trace can be encoded at no extra storage cost; the paper attributes LAGKV's edge over KV-cache baselines to this encode-versus-store distinction.
  • Retrieving more relevant logs (up to k=3 in the paper) generally improves exact match on the tested datasets, making log-store quality and retrieval depth direct levers on accuracy.
  • LAGKV reaches a given accuracy in fewer reasoning steps than a standard agent, so the same compute budget can answer more questions or leave room for additional reasoning on harder ones.
  • Because the store-retrieve-augment loop is described as compatible with any sequential generation workflow, the framework is claimed to extend beyond ReAct-style agents to other LLM generation paradigms.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same stored log could be retrieved at finer granularity than a whole trace, letting a new task reuse only the specific intermediate reasoning step that matches its current sub-question.
  • Editorial inference: the paper's storage analysis implies an adaptive compression policy—store only final-action tokens for easy tasks and more tokens for hard ones—could keep most of the accuracy gain at a fraction of the reported 6.35 GB average store size.
  • Editorial inference: because KV caches are tied to a specific model's hidden-state geometry, a log store built with one open model cannot be consumed by another unless the models share architecture or a projection is learned between KV spaces; the paper does not address cross-model reuse.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes log-augmented generation (LAG), a framework in which reasoning traces from previous tasks are stored as KV caches of tokens from the last model response, retrieved by semantic similarity to the current subtask, and injected into the model's generation after re-rotating positional embeddings. The implementation, LAGKV, is evaluated on Musique, 2WikiMultiHop, GPQA, and MMLU-Pro against a ReAct agent, a reflection baseline, a KV-cache baseline, and text-log variants. The authors report consistent accuracy and efficiency gains, especially on unseen questions, and include ablations on stored token subset and number of retrieved logs.

Significance. If the KV-reuse mechanism is real, LAG offers a genuinely new way to reuse computation across tasks, with practical efficiency gains and accuracy improvements, and the paper's ablations on token subset and number of logs are useful. The work is empirical; the central result is measured on held-out questions and is not circular in the sense of fitting constants to the data. The static log-store setting and the qualitative examples are strengths. However, the reported gains cannot yet be attributed to the semantic content of retrieved reasoning, because no control with irrelevant or random KV vectors is provided. The significance therefore hinges on a single missing experiment.

major comments (4)
  1. [Section 2.2 (Eqs. 2-3), Tables 1-2] The central claim that reusing the retrieved reasoning content causes the accuracy gains is underdetermined because LAGKV is never compared with a content-control condition. All existing baselines differ from LAGKV in more than one way: the KV-cache baseline encodes only the last response without the full reasoning context, LAGtext uses text, and reflection uses extracted insights. None controls for the possibility that injecting any KV vectors—even from an irrelevant log or random tokens—after the approximate RoPE inversion/re-rotation in Eqs. (2)-(3) shifts the model's predictions. A proper control would keep the storage, retrieval, and injection pipeline unchanged but replace the retrieved log KV values with KV values from unrelated logs (or random token sequences) of matched length. Without such a control, the large gains on GPQA unseen (30.4 vs. 18.5 EM) and Musique unseen (32.2 vs. 27.0 EM) cannot be uniquely attributed to the semantic content of the retrieved reasoning.
  2. [Section 3.1, Tables 1-2] The 'seen' evaluation is contaminated. Since the log store is built from the same 70% of questions that are later evaluated as 'seen,' the retriever can retrieve a log of the exact same question, whose last model response contains the answer, making the seen numbers a measure of memorization or in-distribution retrieval rather than generalization. The abstract's claim of 'new, unseen challenges' is supported by the unseen columns, but the seen columns should either be removed or explicitly reframed as an in-distribution memory setting, and the conclusion should not rely on them.
  3. [Section 2.2] The conceptual premise that 'a token's KV value is a weighted aggregation of embeddings from the entire surrounding context' is technically inaccurate for the layer at which the KV is stored. In self-attention, the key and value vectors for a token are produced by linear projections of that token's hidden state; it is the attention output, not the stored KV, that is a weighted sum of value vectors. The token's hidden state is context-dependent through earlier layers, so the claim can be repaired, but as written it misdescribes the mechanism and should be corrected with a precise statement about which quantities are context-dependent.
  4. [Appendix E and Section 3.1] The statistical significance analysis is insufficient. A paired t-test over per-question exact-match outcomes from a single run does not account for run-to-run variance due to sampling in LLM generation and retrieval, and it is unclear whether the t-test was computed on all questions or only those with observed variation. The paper should report means and standard deviations across multiple seeds (or at least bootstrap confidence intervals) for the main comparisons in Tables 1 and 2.
minor comments (5)
  1. [Table 1] There are spacing and formatting errors in the table: '57.565.6' should be '57.5 65.6', and '65.02.29' should be '65.0 2.29'. Similar spacing issues appear in several entries.
  2. [Throughout] The terminology is inconsistent: 'LAG KV', 'LAGKV', 'LAGtext', and 'LAG text' are used interchangeably. Pick one notation per method and use it consistently.
  3. [Figure 3] The x-axis label is rendered as 'Number/uni00A0of/uni00A0max/uni00A0iterations', apparently due to non-breaking spaces; it should read 'Number of max iterations'.
  4. [Table 4] The column 'KV size (GB)' is ambiguous: the values (e.g., 6.1 for 'last action') are difficult to interpret without knowing whether they are per-log, per-question averages, or total store sizes for the 70% split. Please clarify the unit and scope.
  5. [Table 3 and Section 4.1] The definition of 'unsolvable' for Table 3 should be stated more precisely; the text says questions remain unanswered at the performance plateau (iteration 20/25), but it is not specified whether this is a hard cutoff in the experiment or an operational criterion applied post hoc.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LAGKV's reported gains are measured on unseen questions with logs built from a disjoint 70% split, and the central KV-reuse mechanism is an empirical claim rather than an identity or fitted prediction.

full rationale

The derivation chain is empirical and self-contained against external benchmarks. The core claim—that injecting stored KV values from retrieved prior reasoning traces improves accuracy—is tested on Musique, 2WikiMultiHop, GPQA, and MMLU-Pro 'unseen' splits that were not used to construct the log store, so the outcome is not defined by the input representation. No fitted parameters are renamed as predictions; the method's hyperparameters (top-k retrieval, last-round token storage) are choices, and no equation reduces the reported accuracy gain to the method's definitions. The paper contains no load-bearing self-citations: references [7,9] are external prior KV-cache and RoPE-reuse work, and the comparisons are to external methods rather than to the authors' own theorems. The strongest caveats—lack of a control injecting irrelevant or random KV vectors, and selection of the storage strategy using unseen-split analyses—are experimental-validity concerns (plausible alternative explanations, test-set tuning) rather than circularity; they do not make the main result equivalent to its inputs by construction. Score 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method rests on the empirical claim that KV activations compress reasoning context, plus the engineering assumption that re-positioned KV caches transfer across contexts. It introduces no new physical or conceptual entities.

free parameters (3)
  • k (number of retrieved logs) = 3
    Chosen based on test-set comparisons in Table 5; k=3 is used for all main results but is not always the best on every dataset.
  • Stored token subset strategy = Last model response
    Selected from candidates (last 3 rounds, last 2 rounds, last round, last action) using unseen-question performance in Section 4.2; this is tuning on the test set.
  • Maximum reasoning steps C = 8 for multi-hop QA, 3 for GPQA and MMLU-Pro
    Set by task; not central to the method but affects efficiency numbers.
assumptions (3)
  • domain assumption A token's KV value, computed under causal attention, encodes the semantics of the entire preceding context.
    Invoked in Section 2.2 to justify storing only the last response while claiming full context is preserved; empirically plausible but not proven.
  • standard math Stripping RoPE via inverse rotation and reapplying with new position IDs preserves the meaning of the KV values.
    The rotation matrices are orthogonal, so the inverse is mathematically exact, but the assumption that the rest of the KV state is position-independent is a modeling assumption borrowed from prior work [7,9].
  • domain assumption The generation model can correctly interpret KV states that were computed in a different context and position range.
    The entire LAGKV procedure depends on this; no theoretical guarantee is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Log-Augmented Generation: Scaling Test-Time Reasoning with Reusable Computation." pith.science (2026). https://pith.science/paper/45IA3SBG

@misc{pith2026250514398,
  author       = {Pith},
  title        = {Pith review of: Log-Augmented Generation: Scaling Test-Time Reasoning with Reusable Computation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/45IA3SBG}},
  note         = {Machine review of arXiv:2505.14398}
}
read the original abstract

While humans naturally learn and adapt from past experiences, large language models (LLMs) and their agentic counterparts struggle to retain reasoning from previous tasks and apply them in future contexts. To address this limitation, we propose a novel framework, log-augmented generation (LAG) that directly reuses prior computation and reasoning from past logs at test time to enhance model's ability to learn from previous tasks and perform better on new, unseen challenges, all while keeping the system efficient and scalable. Specifically, our system represents task logs using key-value (KV) caches, encoding the full reasoning context of prior tasks while storing KV caches for only a selected subset of tokens. When a new task arises, LAG retrieves the KV values from relevant logs to augment generation. Our approach differs from reflection-based memory mechanisms by directly reusing prior reasoning and computations without requiring additional steps for knowledge extraction or distillation. Our method also goes beyond existing KV caching techniques, which primarily target efficiency gains rather than improving accuracy. Experiments on knowledge- and reasoning-intensive datasets demonstrate that our method significantly outperforms standard agentic systems that do not utilize logs, as well as existing solutions based on reflection and KV cache techniques.

Figures

Figures reproduced from arXiv: 2505.14398 by the authors.

Figure 1
Figure 1. While humans naturally possess the ability to learn from past experiences, LLMs lack this [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. A typical attention weight, after applying the lower-triangular causal mask, enables each [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Exact match performance of standard agentic systems and [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Position: Modular Memory is the Key to Continual Learning Agents

    cs.LG 2026-03 conditional novelty 6.0 of 10

    A modular memory combining in-context learning and in-weight learning is proposed as the key to continual learning agents.

Reference graph

Works this paper leans on

32 extracted references · 18 canonical work pages · cited by 1 Pith paper

  1. [1]

    Memprompt: Memory-assisted prompt editing with user feedback, 2022

    Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. Memprompt: Memory-assisted prompt editing with user feedback, 2022

    work page 2022

  2. [2]

    Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023

    2023

  3. [3]

    Dynamic cheatsheet: Test-time learning with adaptive memory.arXiv preprint arXiv:2504.07952, 2025

    Mirac Suzgun, Mert Yuksekgonul, Federico Bianchi, Dan Jurafsky, and James Zou. Dynamic cheatsheet: Test-time learning with adaptive memory.arXiv preprint arXiv:2504.07952, 2025

    arXiv 2025

  4. [4]

    Thought-retriever: Don’t just retrieve raw data, retrieve thoughts

    Tao Feng, Pengrui Han, Guanyu Lin, Ge Liu, and Jiaxuan You. Thought-retriever: Don’t just retrieve raw data, retrieve thoughts. 2024

    work page 2024

  5. [5]

    Buffer of thoughts: Thought-augmented reasoning with large language models.Advances in Neural Information Processing Systems, 37:113519–113544, 2024

    Ling Yang, Zhaochen Yu, Tianjun Zhang, Shiyi Cao, Minkai Xu, Wentao Zhang, Joseph E Gonzalez, and Bin Cui. Buffer of thoughts: Thought-augmented reasoning with large language models.Advances in Neural Information Processing Systems, 37:113519–113544, 2024

    2024

  6. [6]

    Prompt cache: Modular attention reuse for low-latency inference.Proceedings of Machine Learning and Systems, 6:325–338, 2024

    In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. Prompt cache: Modular attention reuse for low-latency inference.Proceedings of Machine Learning and Systems, 6:325–338, 2024

    2024

  7. [7]

    Turborag: Accelerating retrieval-augmented generation with precomputed kv caches for chunked text.arXiv preprint arXiv:2410.07590, 2024

    Songshuo Lu, Hua Wang, Yutian Rong, Zhi Chen, and Yaohua Tang. Turborag: Accelerating retrieval-augmented generation with precomputed kv caches for chunked text.arXiv preprint arXiv:2410.07590, 2024

    arXiv 2024

  8. [8]

    Don’t do rag: When cache-augmented generation is all you need for knowledge tasks.arXiv preprint arXiv:2412.15605, 2024

    Brian J Chan, Chao-Ting Chen, Jui-Hung Cheng, and Hen-Hsen Huang. Don’t do rag: When cache-augmented generation is all you need for knowledge tasks.arXiv preprint arXiv:2412.15605, 2024

    arXiv 2024

Show all 32 references
  1. [9]

    Block-attention for efficient rag.arXiv preprint arXiv:2409.15355, 2024

    East Sun, Yan Wang, and Lan Tian. Block-attention for efficient rag.arXiv preprint arXiv:2409.15355, 2024

    2024 arXiv

  2. [10]

    Musique: Multihop questions via single-hop question composition.Transactions of the Association for Computational Linguistics, 10:539–554, 2022

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition.Transactions of the Association for Computational Linguistics, 10:539–554, 2022

    2022

  3. [11]

    Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps.arXiv preprint arXiv:2011.01060, 2020

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps.arXiv preprint arXiv:2011.01060, 2020

    2011 arXiv

  4. [12]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. InFirst Conference on Language Modeling, 2024

    2024

  5. [13]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. InThe Thirty-eight Conference on Neural Information Pro...

    2024

  6. [14]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  7. [15]

    Arctic-embed 2.0: Multilingual retrieval without compromise.arXiv preprint arXiv:2412.04506, 2024

    Puxuan Yu, Luke Merrick, Gaurav Nuti, and Daniel Campos. Arctic-embed 2.0: Multilingual retrieval without compromise.arXiv preprint arXiv:2412.04506, 2024

    2024 arXiv

  8. [16]

    Mteb: Massive text embedding benchmark.arXiv preprint arXiv:2210.07316, 2022

    Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. Mteb: Massive text embedding benchmark.arXiv preprint arXiv:2210.07316, 2022. A Prompts The prompts used in the agentic system are shown in Tables 6 and 7. 10 Table 6: Prompt used for knowledge-intensive datasets...

    2022 arXiv

  9. [17]

    From the provided information, extract facts that is relevant to the user question

  10. [18]

    The short answer must be wrapped in <ans></ans>

    Based on the provided information only, determine if you have sufficient information to answer the user question - If you can determine the answer, output a short answer (in a few words) to the user question. The short answer must be wrapped in <ans></ans>. - If you cannot det...

  11. [19]

    From the provided information, extracts the key insights helpful for solving the user question

  12. [20]

    Break down and solve the question step by step, without relying on the provided answer choices

  13. [21]

    - If you cannot identify the answer, output sub-questions that, if solved, can lead to new information

    Based on your analysis, determine if you have sufficient information to identify the single most probable answer - If you can identify the answer, output the answer as the letter corresponding to the answer choice, placed inside parentheses and wrapped in <ans></ans> (e.g., <a...

    1974

  14. [22]

    - The number of stereoisomers can be calculated based on the number of chiral centers (asymmetric carbon atoms) in the molecule

    **Key Insights:** - Stereoisomers are molecules that have the same molecular formula and sequence of bonded atoms but differ in the three-dimensional orientations of their atoms in space. - The number of stereoisomers can be calculated based on the number of chiral centers (as...

  15. [23]

    - To find the number of stereoisomers, we need to identify the chiral centers in the molecule

    **Breaking Down the Problem:** - The given compound is 6-chloro-9-ethyl-2-methylundeca-3,7-dien-5-ol. - To find the number of stereoisomers, we need to identify the chiral centers in the molecule. - A chiral center is an asymmetric carbon atom with four different groups attach...

  16. [24]

    - We can break down the molecule into its components: - The main chain has 11 carbon atoms

    **Analyzing the Molecule:** - The molecule has the following structure: 6-chloro-9-ethyl-2-methylundeca-3,7-dien-5-ol. - We can break down the molecule into its components: - The main chain has 11 carbon atoms. - There are two double bonds at positions 3 and 7. - There is a ch...

  17. [25]

    - Each chiral center can have two possible configurations (R or S)

    **Calculating the Number of Stereoisomers:** - We have identified three chiral centers in the molecule. - Each chiral center can have two possible configurations (R or S). - Therefore, the total number of stereoisomers is2 3 = 8. Based on our analysis, we can conclude that the...

  18. [26]

    (Z)-1-chloro-2-methylbut-1-ene: This compound is an alkene with a (Z) configuration, which means it has a specific arrangement of atoms but is notchiral... (...omitted logs...) (User prompt) (...omitted instructions...) How many stereoisomers are there for the compound 6-chlor...

  19. [27]

    This suggests that it may have multiple stereocenters

    The compound 6-chloro-9-ethyl-2-methylundeca-3,7-dien-5-ol has a complex structure with multiple double bonds and a hydroxyl group. This suggests that it may have multiple stereocenters. 2.The compound has three double bonds (3, 7, and 5), which means it has the potential for ...

  20. [28]

    The compound also has a hydroxyl group (5-ol), which means it has a stereocenter at the 5-position

  21. [29]

    To determine the number of stereoisomers, we need to consider the number of stereocenters and the possible configurations at each stereocenter

  22. [30]

    The compound has 3double bonds, which can result in 23 = 8possibleE/Z configura- tions

  23. [31]

    The compound has 1 stereocenter (5-ol), which can have 2 possible configurations (R and S)

  24. [32]

    To calculate the total number of stereoisomers, we need to multiply the number of E/Z configurations by the number of possible configurations at the stereocenter. 8. The total num- ber of stereoisomers is 8 (E/Z configurations) x 2 (possible configurations at the stereocenter)...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.