Pith.

Artificial Intelligence · arXiv:2610.19474 v2 · submitted 14 Oct 2026

When Agent Memory Needs to Compute: Evidence Access and Executable Aggregation

Zerui Yang, Quan Xue, Chaorui Zhang, Xueyan Niu, Xuanyu Zheng, Xiyuan Wang, Haolin Ruan, Guangcheng Song

show 3 more authors

Weixi Zhang, Wei Han, Bo Bai

The paper says

“Questions about an agent's history often require aggregating events, which retrieval alone does not resolve. We combine source-preserving evidence access with executable aggregation. Trajectory Memory (TM) retains the original trajectory and constructs a source-addressable reading view at question time.”

more of the abstract

TallyMem augments it with a reusable event ledger, model-generated event specifications and deterministic execution. With a commit-specific adaptation on 776 questions from 99 held-out MINTEval repositories, TallyMem raises Qwen3-32B exact match from 35.2% to 49.0%, a 13.8-point gain over TM; gains span 12.8-13.9 points across three readers. A controlled probe with event sets parsed independently from raw logs shows an 11.0-point gain, from 82.8% to 93.8%, without routing. On full AMA-Bench, the routed system yields a smaller 1.3-point gain, from 69.8% to 71.1%. These results support executable aggregation on tasks that require combining events, with benefits dependent on the task and event specification, the largest residual system-error category in our diagnostic analysis.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 15 Oct 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up. Honest, well-executed system for separating memory access from event aggregation, with clear limitation that specification errors remain the main bottleneck. This line is wrong

2 major objections · 7 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

The paper says, at Section 7.4 (Controlled Aggregation Probe)

The review did not quote this passage.

Pith says: Objection 1 of 2 · major

The probe templates cover exactly the six operations (count, list, first, last, group, distinct) that TallyMem's query language supports, and they use step bounds and event predicates that align with the ledger's field schema (tool, action, target, outcome). This design necessarily favors the proposed method over systems without explicit aggregation.

more of objection 1

The paper acknowledges this indirectly in the limitations (Appendix H), but Sections 7.4 and the main text would benefit from a sentence stating directly that the probe was designed to test aggregation under conditions where the event sets and operations match the ledger's representational capacity, so the strong results (11.0-point gain, correct-plan arm at 94.9%) show an upper bound on what the mechanism can achieve when specification is accurate, rather than a claim of equal advantage on real benchmark questions.

The paper says, at Section 7.3 and Appendix M (MINTEval adaptation)

The review did not quote this passage.

Pith says: Objection 2 of 2 · major

The MINTEval evaluation adapts the ledger schema and compiler prompt to commit diffs: 'This is a benchmark-specific adaptation of the interface, not zero-adaptation transfer' (Appendix M). The 13.8-point gain is therefore a demonstration of the framework's adaptability rather than a zero-shot transfer result.

more of objection 2

The abstract and introduction state 'commit-specific adaptation,' which is accurate, but the paper does not quantify the adaptation effort—e.g., the number of prompt iterations or development repositories used (25 given in Section 7.3). Reporting these details would help readers assess the generality of the approach to new domains.

7 minor objections

The paper says, at Section 7.2

Pith says: minor objection 1 of 7

The sentence 'TM also exceeds the external memory systems in the same Qwen3-32B deployment, of which BM25 is strongest at 46.1%' is correct but slightly misleading because the external systems use much smaller context budgets (e.g., BM25 shows the five top-scoring blocks, much less than TM's 26k-token view). Qualify the comparison or move it to a paragraph that notes the budget asymmetry.

The paper says, at Figure 2 (left) and Section 5

Pith says: minor objection 2 of 7

The compiler-defined load bins in Figure 2 are described as 'descriptive diagnostics, not independent difficulty strata' (Section 5). This caveat should appear in the figure caption as well, not only in the body text, to prevent misinterpretation by readers who scan figures.

show 5 more

The paper says, at Table 3

Pith says: minor objection 3 of 7

The 'Other readers (%)' column shows the accuracy of the other four readers given the same evidence. The text explains this in Section 8, but the table caption itself does not define the column. Add a brief caption note, e.g., 'Accuracy of the other four readers (Qwen3-235B, gpt-oss-120b, Qwen3.6-27B, Qwen3-8B) on the same evidence.'

The paper says, at Appendix H

Pith says: minor objection 4 of 7

The correct-plan arm was 'designed after inspecting TallyMem's misses on S3–S5, so it is diagnostic rather than confirmatory' (Appendix H). This is transparent, but the main text (Section 7.4) states only that the arm was added after results; it should note the post-hoc nature to avoid giving the impression that the 94.9% result was pre-registered.

The paper says, at Appendix J (Router Definition)

Pith says: minor objection 5 of 7

The router uses γ_load on AMA-Bench and γ_cov on MINTEval, where γ_cov adds a coverage condition. The difference is stated but not motivated. A brief explanation of why coverage routing is helpful on MINTEval (e.g., because TM trails long context there and coverage is lower) would help.

The paper says, at General

Pith says: minor objection 6 of 7

The paper uses 'long-context reading' to refer to a specific truncation strategy (70:30 head-tail split, Appendix M). In Section 7.1 the setup for long context should explicitly mention this split, not just 'truncation to the budget.'

The paper says, at Section 9 and Limitations

Pith says: minor objection 7 of 7

The limitation that 'the ledger and the query language cannot express every event predicate' is stated, but the paper does not give examples of predicates that cannot be expressed. Adding one or two concrete examples (e.g., temporal ordering between two event types that the language's conjunction/disjunction cannot capture, or arithmetic on numeric values) would make the limitation more informative.

What would settle it

Re-run the MINTEval evaluation with TM's retrieval budget increased to include all commits (e.g., B=100k tokens instead of 26k), then measure TallyMem's gain over this stronger TM baseline. If the gain shrinks substantially (e.g., from ~14 points to <5 points), the MINTEval result is primarily an access benefit rather than an aggregation benefit; if the gain remains large, it strengthens the aggregation claim. This test isolates the two mechanisms that the paper already distinguishes in its diagnosis.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 15 Oct 2026

  1. Reader holds up confidence high
  2. Skeptic agrees 24 s · 33,530 tokens read, 2,590 written
  3. Editor wrote the verdict above 19 s · 34,389 tokens read, 1,380 written
  4. Referee accept: 2 major objections, 7 minor 47 s · 33,904 tokens read, 3,448 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

This paper does something the field needs: it pulls apart two distinct failure modes in agent memory—access (can the relevant evidence reach the reader?) and aggregation (can the reader combine multiple events correctly?)—and builds complementary mechanisms for each. TM keeps the trajectory verbatim and selects a budgeted reading view at question time; TallyMem compiles event specifications (scope, predicate, operation) into a typed query language and executes them deterministically over a reusable ledger.

The evaluation is thorough and unusually honest. The controlled probe cleanly isolates the aggregation mechanism, showing an 11-point gain with known event sets. The held-out MINTEval evaluation shows a 13.8-point gain, though that comes with a benchmark-specific adaptation. The full AMA-Bench gain is only 1.3 points, but the paper reports it plainly, shows it concentrates on high-load questions, and attributes the limited benefit to specification errors.

The main soft spot is right where the paper says it is: the model must write correct event specifications, and the error analysis shows that specification errors are the largest residual category (49% of system errors). The approach shifts the hard part from counting to specifying what to count, and the model isn't always reliable at that. The MINTEval adaptation is domain-specific; the controlled probe uses generated templates rather than natural questions. The free parameters (token budget, per-field caps, anchor weights) are listed but not deeply ablated.

These are real limitations, but the paper owns them. The central argument—that separating event specification from deterministic execution reduces aggregation errors—is well-supported where it can be tested. The error diagnostics make the residual problem inspectable, which is itself a contribution.

This paper deserves peer review. It is a solid system paper with honest evaluation. I would cite it if working on agent memory.

The strongest objection

The paper's central claim — that separating event specification from deterministic execution reduces aggregation errors — is well-supported by its evidence and honestly bounded by its limitations. The weakest assumption identified by the reader (model-generated specifications) is extensively analyzed: Section 8 and Table 3 show specification errors are the largest residual category (51/148, 49% of system errors), and the paper explicitly states this as 'the main residual challenge.' The controlled probe (Section 7.4) isolates the aggregation mechanism cleanly, showing an 11.0-point gain with known event sets, and the correct-plan arm (Appendix H) reaches 94.9%, confirming the execution pathway works when specifications are right. The full AMA-Bench gain is modest (1.3 points, not significant without routing), but the paper reports this honestly and attributes the limited benefit to specification errors. On MINTEval, the 13.8-point gain is partly an access advantage (TM trails long context because its retrieval budget loses evidence), but the paper acknowledges this explicitly in Section 7.3. No internal inconsistency or hidden assumption undermines the argument; the paper's own diagnostics supply the relevant caveats.

The reader identified model-generated specifications as the weakest assumption, which is exactly the paper's own main finding from its error analysis. The paper provides multiple lines of converging evidence (controlled probe, MINTEval held-out, correct-plan arm, free-form code comparison) that the execution mechanism works when specifications are correct. The specification bottleneck is documented and quantified. I see no unaddressed concern that would change the verdict. The paper's conclusions are appropriately hedged, and the limitations section transparently covers scope, load-proxy caveats, and the single-annotator error analysis. The recommended concrete test would help sharpen the attribution of MINTEval gains but does not undermine the current conclusions.

The referee's report

If the results are sustained, the paper makes a solid and well-bounded contribution. It cleanly distinguishes evidence access from event aggregation through controlled experiments and a multi-reader diagnostic, and it provides an executable aggregation mechanism that makes event selection explicit and errors inspectable, in contrast to black-box reader miscounts. The controlled probe with independently parsed event sets is a methodological strength that isolates the aggregation mechanism. The paper's honesty about its modest full-benchmark gain, its benchmark-specific MINTEval adaptation, and its identification of specification errors as the main residual challenge sets a good standard for empirical work in this area.

Pith's reading of the paper (a machine's summary, not the authors')

Questions about an agent's history often require counting or grouping events, a task at which even long-context readers miscount. The authors identify two separable bottlenecks: evidence access (does the reader see the relevant steps?) and event aggregation (can it combine them correctly?). They first build Trajectory Memory (TM), which preserves the trajectory verbatim and selects what to show only at question time. On top of it they build TallyMem, which has a model write an explicit event specification (a scope, a predicate, and an operation) and a deterministic executor run that specification over a pre-built event ledger. This converts aggregation from a model task into a specification task: the model decides what to count, but the counting is exact. The gains are large on held-out repository histories (13.8 points) and a controlled probe (11.0 points), but modest on the full AMA-Bench benchmark (1.3 points). The paper's diagnostic analysis shows that the largest residual error category is specification—the model writes a plan that selects the wrong events—not execution or ledger quality. The architecture thus trades miscounting (which leaves no audit trail) for misspecification (which produces a query and result that can be inspected against the log).

What carries the argument. TallyMem's central mechanism is separating event specification from execution via (1) a pre-built event ledger L with one row per trajectory step (tool, action, targets, outcome, changes) and (2) a typed query language in which a model writes a plan π = (I, φ, ω). The executor grounds each filter against the ledger, selectes rows that match the predicate φ within scope I, and runs the operation ω (count, list, first/last, group-count, timeline) deterministically. This makes aggregation exact when the specification matches the intended events, and makes specification errors visibile because the plan, filters, and step IDs are returned as evidence alongside the computed values.

Core claim. TallyMem improves aggregation accuracy by separating event specification from event execution. A language model writes a typed query plan π = (I, φ, ω) that names a step scope, an event predicate (a conjunction of field filters), and an aggregation operation (count, list, first/last, group-count, or timeline). A fixed executor evaluates the plan deterministically over a reusable event ledger L that records one row per trajectory step (tool, action, targets, outcome, changes, parameter difference). The model never does the counting; it only decides which events to count. On the MINTEval GitHub subset (776 questions from 99 held-out repositories), this raises exact match from 35.2% (TM) to 49.0% (TallyMem). On a controlled probe of 645 questions with event sets parsed independently from raw logs, accuracy goes from 82.8% to 93.8%. The gain on the full AMA-Bench (2,496 questions) is smaller: from 69.8% to 71.1%. The paper argues that the mechanism works well when questions require combining many events, and that the remaining errors are dominated by specification mistakes (49% of system errors), which are inspecticianable because each result states its filters and supporting step IDs.

The premise it rests on. The language model can consistently write correct event specifications (scope, predicate, operation) for each question, aligning its notion of what counts as an event with the reference answer's notion.

If the paper is right (Pith's inferences, not the authors')

Agent memory systems built on retrieval alone should augment themselves with an executable aggregation layer for any question type that involves counts, lists, timeliness, or group-by operations over multiple steps.

The approach converts the untraceable failure of miscounting (e.g., a reader listing 18 'examine' events instead of 5) into the traceable failure of misspecician (a plan that filters on the wrong predicate); this makes root-cause diagnosis possible from the output evidence alone.

Because the event ledger is built question-independently from the raw trajectory, the same ledger supports many aggregation questions across the entire history; no per-question memory construction is needed.

On tasks where questions reliably name the events they mean (e.g., 'how many times did the agent open drawer 1?'), the routed system's gain is largest; on tasks with ambiguous or underspecified event references, specification errors dominate and limit the benefit.

The paper's diagnostic method—cross-tabulating aggregation load against context fit—provides a general template for evaluating any agent memory system, separating whether failures are due to missing evidence or difficulty in combining the evidence that is present.

If the core practitioner takeaway is that executable aggregation is cost-effective when and only when the specification bottleneck can be cheaply addressed, the paper's fast that a fixed counter (a single hardcoded query) recovers about half the gain on the probe suggests that even very simple executable aggregate can outperform a language-model reader at high aggregation loads.

The paper's architecture mirrors the classic database separation of concern: a query optimizer (the LLM compiler) produces a plan, and a query executor runs it. This isomorphism suggests that decades of database research on query optimization, cost-based plan selection, and view materialization could productively be ported to agent memory.

If the specification bottleneck is the dominant remaining error source, then the path to higher accuracy is not better execution or longer contexts but better alignment between the compiler's event model and the question writer's event model. A learned classifier that maps question phrases to predicates crowdsourced from a few examples might close mostof the remaining gap.

The large gain on the controlled probe (11.0 points) relative to the small gain on full AMA-Bench (1.3 points) suggests that many AMA-Bench questions either (a) do not require aggregation (low load), or (b) use event references the compiler cannot reliably expression (e.g., 'which containers...?' maps to a different predicate than the reference answer expects). Benchmarks that mix access-bound and

The paper's 'correct-plan arm' on the probe (99.2% on expressible templates) implies that a perfect compiler would recover almost all aggregation errors. Designing a language-level interface—perhaps natural-language-to-query for agent histories with strong type constraints—could be the most direct follow-up.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Questions about an agent's history often require aggregating events, which retrieval alone does not resolve. We combine source-preserving evidence access with executable aggregation. Trajectory Memory (TM) retains the original trajectory and constructs a source-addressable reading view at question time. TallyMem augments it with a reusable event ledger, model-generated event specifications and deterministic execution. With a commit-specific adaptation on 776 questions from 99 held-out MINTEval repositories, TallyMem raises Qwen3-32B exact match from 35.2% to 49.0%, a 13.8-point gain over TM; gains span 12.8-13.9 points across three readers. A controlled probe with event sets parsed independently from raw logs shows an 11.0-point gain, from 82.8% to 93.8%, without routing. On full AMA-Bench, the routed system yields a smaller 1.3-point gain, from 69.8% to 71.1%. These results support executable aggregation on tasks that require combining events, with benefits dependent on the task and event specification, the largest residual system-error category in our diagnostic analysis.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-15

The authors' main claim, as the classifier picked it out 1.00

“These results support executable aggregation on tasks that require combining events, with benefits dependent on the task and event specification, the largest residual system-error category in our diagnostic analysis.”

How the main claim is stated: asserted 0.78

What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.91 · human participants 0.05 · controlled comparison 0.92 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.97 · mentions code or data availability 0.04

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Questions about an agent's history often require aggregating events, which retrieval alone does not resolve. background 0.73 0.10 0.06 asserted 0.97
We combine source-preserving evidence access with executable aggregation. contribution 0.87 0.87 0.10 asserted 0.98
Trajectory Memory (TM) retains the original trajectory and constructs a source-addressable reading view at question time. method 0.88 0.76 0.05 asserted 1.00
TallyMem augments it with a reusable event ledger, model-generated event specifications and deterministic execution. method 0.87 0.84 0.05 asserted 0.97
With a commit-specific adaptation on 776 questions from 99 held-out MINTEval repositories, TallyMem raises Qwen3-32B exact match from 35.2% to 49.0%, a 13.8-point gain over TM; gains span 12.8-13.9 points across three readers. result 1.00 0.96 0.17 asserted 1.00
A controlled probe with event sets parsed independently from raw logs shows an 11.0-point gain, from 82.8% to 93.8%, without routing. result 1.00 0.94 0.06 asserted 1.00
On full AMA-Bench, the routed system yields a smaller 1.3-point gain, from 69.8% to 71.1%. result 0.99 0.94 0.06 asserted 1.00
These results support executable aggregation on tasks that require combining events, with benefits dependent on the task and event specification, the largest residual system-error category in our diagnostic analysis. interpretation 1.00 0.93 0.83 asserted 0.78

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.96

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-15: 1 request, 6,294 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (12)
Figure 1 of the paper
Figure 1. Evidence access and event aggregation on an action-frequency question. Left: an illustrative compression
Figure 2 of the paper
Figure 2. Compiler-defined load diagnostic and log
Figure 3 of the paper
Figure 3. Two constraints on trajectory answering.
Figure 4 of the paper
Figure 4. TM and TallyMem through a shared running example. (a) TM retrieves source text from archive
Figure 5 of the paper
Figure 5. TM’s archive and reading view (illustrative records). Deterministic indexes guide budgeted selection; the
Figure 6 of the paper
Figure 6. Comparison of Qwen3-32B systems on the 2,496 AMA-Bench questions. TallyMem denotes the routed main system; “no routing” is its ablation. Ex￾ternal memory systems use the same deployment, with failed memory constructions counted as wrong. Deploy￾ment and adaptation details below define the comparison scope. exceeds long context (+394/−315, p = 0.003) and TallyMem exceeds TM (+50/−12, p < 0.001). Published AMA scores use different settings: their Qwen3 long-context control is 51.8%, compared with 65.0% here, and their AMA-Agent scores 57.2%. These scores provide context; matched comp
Figure 7 of the paper
Figure 7. Five-reader diagnostic comparison, without routing: (a) all 630 common rule-selected questions; (b) the
Figure 8 of the paper
Figure 8. MINTEval value-history questions by com
Figure 9 of the paper
Figure 9. A conceptual example of Equation 12. The executed set includes an extra event and misses an intended one. Their effects cancel in the scalar count, although the selected sets differ. S ⋆ is an analysis reference, not an input to inference. Rendering and final answer generation can introduce additional errors. Domain n Long ctx. TM TallyMem-lite TallyMem TallyMem vs. TM Embodied 239 129 141 161 165 +33/−9 (<0.001) Game 17 12 10 10 8 +0/−2 (0.50) Open-world 59 30 45 45 43 +0/−2 (0.50) Software 67 27 21 21 20 +3/−4 (1.00) Text-to-SQL 153 97 99 96 107 +16/−8 (0.15) Web 95 55 74 74 73…
Figure 10 of the paper
Figure 10. Computed evidence Dq: tool frequencies, supporting step IDs and outcome breakdowns (excerpt). Saved answers
Figure 11 of the paper
Figure 11. Reading and execution on the worked example. Left: TM’s reading view and the full-ledger computation
Figure 12 of the paper
Figure 12. Saved answers across reader configurations on the same question. Red entries differ from the reference;

Figures are the authors', reproduced from the paper.

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment