Artificial Intelligence · arXiv:2610.19474 v2 · submitted 14 Oct 2026
When Agent Memory Needs to Compute: Evidence Access and Executable Aggregation
The paper says
“Questions about an agent's history often require aggregating events, which retrieval alone does not resolve. We combine source-preserving evidence access with executable aggregation. Trajectory Memory (TM) retains the original trajectory and constructs a source-addressable reading view at question time.”
more of the abstract
TallyMem augments it with a reusable event ledger, model-generated event specifications and deterministic execution. With a commit-specific adaptation on 776 questions from 99 held-out MINTEval repositories, TallyMem raises Qwen3-32B exact match from 35.2% to 49.0%, a 13.8-point gain over TM; gains span 12.8-13.9 points across three readers. A controlled probe with event sets parsed independently from raw logs shows an 11.0-point gain, from 82.8% to 93.8%, without routing. On full AMA-Bench, the routed system yields a smaller 1.3-point gain, from 69.8% to 71.1%. These results support executable aggregation on tasks that require combining events, with benefits dependent on the task and event specification, the largest residual system-error category in our diagnostic analysis.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 15 Oct 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up. Honest, well-executed system for separating memory access from event aggregation, with clear limitation that specification errors remain the main bottleneck. This line is wrong
2 major objections · 7 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
The paper says, at Section 7.4 (Controlled Aggregation Probe)
The review did not quote this passage.
Pith says: Objection 1 of 2 · major
The probe templates cover exactly the six operations (count, list, first, last, group, distinct) that TallyMem's query language supports, and they use step bounds and event predicates that align with the ledger's field schema (tool, action, target, outcome). This design necessarily favors the proposed method over systems without explicit aggregation.
more of objection 1
The paper acknowledges this indirectly in the limitations (Appendix H), but Sections 7.4 and the main text would benefit from a sentence stating directly that the probe was designed to test aggregation under conditions where the event sets and operations match the ledger's representational capacity, so the strong results (11.0-point gain, correct-plan arm at 94.9%) show an upper bound on what the mechanism can achieve when specification is accurate, rather than a claim of equal advantage on real benchmark questions.
The paper says, at Section 7.3 and Appendix M (MINTEval adaptation)
The review did not quote this passage.
Pith says: Objection 2 of 2 · major
The MINTEval evaluation adapts the ledger schema and compiler prompt to commit diffs: 'This is a benchmark-specific adaptation of the interface, not zero-adaptation transfer' (Appendix M). The 13.8-point gain is therefore a demonstration of the framework's adaptability rather than a zero-shot transfer result.
more of objection 2
The abstract and introduction state 'commit-specific adaptation,' which is accurate, but the paper does not quantify the adaptation effort—e.g., the number of prompt iterations or development repositories used (25 given in Section 7.3). Reporting these details would help readers assess the generality of the approach to new domains.
7 minor objections
The paper says, at Section 7.2
Pith says: minor objection 1 of 7
The sentence 'TM also exceeds the external memory systems in the same Qwen3-32B deployment, of which BM25 is strongest at 46.1%' is correct but slightly misleading because the external systems use much smaller context budgets (e.g., BM25 shows the five top-scoring blocks, much less than TM's 26k-token view). Qualify the comparison or move it to a paragraph that notes the budget asymmetry.
The paper says, at Figure 2 (left) and Section 5
Pith says: minor objection 2 of 7
The compiler-defined load bins in Figure 2 are described as 'descriptive diagnostics, not independent difficulty strata' (Section 5). This caveat should appear in the figure caption as well, not only in the body text, to prevent misinterpretation by readers who scan figures.
show 5 more
The paper says, at Table 3
Pith says: minor objection 3 of 7
The 'Other readers (%)' column shows the accuracy of the other four readers given the same evidence. The text explains this in Section 8, but the table caption itself does not define the column. Add a brief caption note, e.g., 'Accuracy of the other four readers (Qwen3-235B, gpt-oss-120b, Qwen3.6-27B, Qwen3-8B) on the same evidence.'
The paper says, at Appendix H
Pith says: minor objection 4 of 7
The correct-plan arm was 'designed after inspecting TallyMem's misses on S3–S5, so it is diagnostic rather than confirmatory' (Appendix H). This is transparent, but the main text (Section 7.4) states only that the arm was added after results; it should note the post-hoc nature to avoid giving the impression that the 94.9% result was pre-registered.
The paper says, at Appendix J (Router Definition)
Pith says: minor objection 5 of 7
The router uses γ_load on AMA-Bench and γ_cov on MINTEval, where γ_cov adds a coverage condition. The difference is stated but not motivated. A brief explanation of why coverage routing is helpful on MINTEval (e.g., because TM trails long context there and coverage is lower) would help.
The paper says, at General
Pith says: minor objection 6 of 7
The paper uses 'long-context reading' to refer to a specific truncation strategy (70:30 head-tail split, Appendix M). In Section 7.1 the setup for long context should explicitly mention this split, not just 'truncation to the budget.'
The paper says, at Section 9 and Limitations
Pith says: minor objection 7 of 7
The limitation that 'the ledger and the query language cannot express every event predicate' is stated, but the paper does not give examples of predicates that cannot be expressed. Adding one or two concrete examples (e.g., temporal ordering between two event types that the language's conjunction/disjunction cannot capture, or arithmetic on numeric values) would make the limitation more informative.
What would settle it
Re-run the MINTEval evaluation with TM's retrieval budget increased to include all commits (e.g., B=100k tokens instead of 26k), then measure TallyMem's gain over this stronger TM baseline. If the gain shrinks substantially (e.g., from ~14 points to <5 points), the MINTEval result is primarily an access benefit rather than an aggregation benefit; if the gain remains large, it strengthens the aggregation claim. This test isolates the two mechanisms that the paper already distinguishes in its diagnosis.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 15 Oct 2026
- Reader holds up
- Skeptic agrees
- Editor wrote the verdict above
- Referee accept: 2 major objections, 7 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
This paper does something the field needs: it pulls apart two distinct failure modes in agent memory—access (can the relevant evidence reach the reader?) and aggregation (can the reader combine multiple events correctly?)—and builds complementary mechanisms for each. TM keeps the trajectory verbatim and selects a budgeted reading view at question time; TallyMem compiles event specifications (scope, predicate, operation) into a typed query language and executes them deterministically over a reusable ledger.
The evaluation is thorough and unusually honest. The controlled probe cleanly isolates the aggregation mechanism, showing an 11-point gain with known event sets. The held-out MINTEval evaluation shows a 13.8-point gain, though that comes with a benchmark-specific adaptation. The full AMA-Bench gain is only 1.3 points, but the paper reports it plainly, shows it concentrates on high-load questions, and attributes the limited benefit to specification errors.
The main soft spot is right where the paper says it is: the model must write correct event specifications, and the error analysis shows that specification errors are the largest residual category (49% of system errors). The approach shifts the hard part from counting to specifying what to count, and the model isn't always reliable at that. The MINTEval adaptation is domain-specific; the controlled probe uses generated templates rather than natural questions. The free parameters (token budget, per-field caps, anchor weights) are listed but not deeply ablated.
These are real limitations, but the paper owns them. The central argument—that separating event specification from deterministic execution reduces aggregation errors—is well-supported where it can be tested. The error diagnostics make the residual problem inspectable, which is itself a contribution.
This paper deserves peer review. It is a solid system paper with honest evaluation. I would cite it if working on agent memory.
The strongest objection
The paper's central claim — that separating event specification from deterministic execution reduces aggregation errors — is well-supported by its evidence and honestly bounded by its limitations. The weakest assumption identified by the reader (model-generated specifications) is extensively analyzed: Section 8 and Table 3 show specification errors are the largest residual category (51/148, 49% of system errors), and the paper explicitly states this as 'the main residual challenge.' The controlled probe (Section 7.4) isolates the aggregation mechanism cleanly, showing an 11.0-point gain with known event sets, and the correct-plan arm (Appendix H) reaches 94.9%, confirming the execution pathway works when specifications are right. The full AMA-Bench gain is modest (1.3 points, not significant without routing), but the paper reports this honestly and attributes the limited benefit to specification errors. On MINTEval, the 13.8-point gain is partly an access advantage (TM trails long context because its retrieval budget loses evidence), but the paper acknowledges this explicitly in Section 7.3. No internal inconsistency or hidden assumption undermines the argument; the paper's own diagnostics supply the relevant caveats.
The reader identified model-generated specifications as the weakest assumption, which is exactly the paper's own main finding from its error analysis. The paper provides multiple lines of converging evidence (controlled probe, MINTEval held-out, correct-plan arm, free-form code comparison) that the execution mechanism works when specifications are correct. The specification bottleneck is documented and quantified. I see no unaddressed concern that would change the verdict. The paper's conclusions are appropriately hedged, and the limitations section transparently covers scope, load-proxy caveats, and the single-annotator error analysis. The recommended concrete test would help sharpen the attribution of MINTEval gains but does not undermine the current conclusions.
The referee's report
If the results are sustained, the paper makes a solid and well-bounded contribution. It cleanly distinguishes evidence access from event aggregation through controlled experiments and a multi-reader diagnostic, and it provides an executable aggregation mechanism that makes event selection explicit and errors inspectable, in contrast to black-box reader miscounts. The controlled probe with independently parsed event sets is a methodological strength that isolates the aggregation mechanism. The paper's honesty about its modest full-benchmark gain, its benchmark-specific MINTEval adaptation, and its identification of specification errors as the main residual challenge sets a good standard for empirical work in this area.
Pith's reading of the paper (a machine's summary, not the authors')
Questions about an agent's history often require counting or grouping events, a task at which even long-context readers miscount. The authors identify two separable bottlenecks: evidence access (does the reader see the relevant steps?) and event aggregation (can it combine them correctly?). They first build Trajectory Memory (TM), which preserves the trajectory verbatim and selects what to show only at question time. On top of it they build TallyMem, which has a model write an explicit event specification (a scope, a predicate, and an operation) and a deterministic executor run that specification over a pre-built event ledger. This converts aggregation from a model task into a specification task: the model decides what to count, but the counting is exact. The gains are large on held-out repository histories (13.8 points) and a controlled probe (11.0 points), but modest on the full AMA-Bench benchmark (1.3 points). The paper's diagnostic analysis shows that the largest residual error category is specification—the model writes a plan that selects the wrong events—not execution or ledger quality. The architecture thus trades miscounting (which leaves no audit trail) for misspecification (which produces a query and result that can be inspected against the log).
What carries the argument. TallyMem's central mechanism is separating event specification from execution via (1) a pre-built event ledger L with one row per trajectory step (tool, action, targets, outcome, changes) and (2) a typed query language in which a model writes a plan π = (I, φ, ω). The executor grounds each filter against the ledger, selectes rows that match the predicate φ within scope I, and runs the operation ω (count, list, first/last, group-count, timeline) deterministically. This makes aggregation exact when the specification matches the intended events, and makes specification errors visibile because the plan, filters, and step IDs are returned as evidence alongside the computed values.
Core claim. TallyMem improves aggregation accuracy by separating event specification from event execution. A language model writes a typed query plan π = (I, φ, ω) that names a step scope, an event predicate (a conjunction of field filters), and an aggregation operation (count, list, first/last, group-count, or timeline). A fixed executor evaluates the plan deterministically over a reusable event ledger L that records one row per trajectory step (tool, action, targets, outcome, changes, parameter difference). The model never does the counting; it only decides which events to count. On the MINTEval GitHub subset (776 questions from 99 held-out repositories), this raises exact match from 35.2% (TM) to 49.0% (TallyMem). On a controlled probe of 645 questions with event sets parsed independently from raw logs, accuracy goes from 82.8% to 93.8%. The gain on the full AMA-Bench (2,496 questions) is smaller: from 69.8% to 71.1%. The paper argues that the mechanism works well when questions require combining many events, and that the remaining errors are dominated by specification mistakes (49% of system errors), which are inspecticianable because each result states its filters and supporting step IDs.
The premise it rests on. The language model can consistently write correct event specifications (scope, predicate, operation) for each question, aligning its notion of what counts as an event with the reference answer's notion.
If the paper is right (Pith's inferences, not the authors')
Agent memory systems built on retrieval alone should augment themselves with an executable aggregation layer for any question type that involves counts, lists, timeliness, or group-by operations over multiple steps.
The approach converts the untraceable failure of miscounting (e.g., a reader listing 18 'examine' events instead of 5) into the traceable failure of misspecician (a plan that filters on the wrong predicate); this makes root-cause diagnosis possible from the output evidence alone.
Because the event ledger is built question-independently from the raw trajectory, the same ledger supports many aggregation questions across the entire history; no per-question memory construction is needed.
On tasks where questions reliably name the events they mean (e.g., 'how many times did the agent open drawer 1?'), the routed system's gain is largest; on tasks with ambiguous or underspecified event references, specification errors dominate and limit the benefit.
The paper's diagnostic method—cross-tabulating aggregation load against context fit—provides a general template for evaluating any agent memory system, separating whether failures are due to missing evidence or difficulty in combining the evidence that is present.
If the core practitioner takeaway is that executable aggregation is cost-effective when and only when the specification bottleneck can be cheaply addressed, the paper's fast that a fixed counter (a single hardcoded query) recovers about half the gain on the probe suggests that even very simple executable aggregate can outperform a language-model reader at high aggregation loads.
The paper's architecture mirrors the classic database separation of concern: a query optimizer (the LLM compiler) produces a plan, and a query executor runs it. This isomorphism suggests that decades of database research on query optimization, cost-based plan selection, and view materialization could productively be ported to agent memory.
If the specification bottleneck is the dominant remaining error source, then the path to higher accuracy is not better execution or longer contexts but better alignment between the compiler's event model and the question writer's event model. A learned classifier that maps question phrases to predicates crowdsourced from a few examples might close mostof the remaining gap.
The large gain on the controlled probe (11.0 points) relative to the small gain on full AMA-Bench (1.3 points) suggests that many AMA-Bench questions either (a) do not require aggregation (low load), or (b) use event references the compiler cannot reliably expression (e.g., 'which containers...?' maps to a different predicate than the reference answer expects). Benchmarks that mix access-bound and
The paper's 'correct-plan arm' on the probe (99.2% on expressible templates) implies that a perfect compiler would recover almost all aggregation errors. Designing a language-level interface—perhaps natural-language-to-query for agent histories with strong type constraints—could be the most direct follow-up.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Questions about an agent's history often require aggregating events, which retrieval alone does not resolve. We combine source-preserving evidence access with executable aggregation. Trajectory Memory (TM) retains the original trajectory and constructs a source-addressable reading view at question time. TallyMem augments it with a reusable event ledger, model-generated event specifications and deterministic execution. With a commit-specific adaptation on 776 questions from 99 held-out MINTEval repositories, TallyMem raises Qwen3-32B exact match from 35.2% to 49.0%, a 13.8-point gain over TM; gains span 12.8-13.9 points across three readers. A controlled probe with event sets parsed independently from raw logs shows an 11.0-point gain, from 82.8% to 93.8%, without routing. On full AMA-Bench, the routed system yields a smaller 1.3-point gain, from 69.8% to 71.1%. These results support executable aggregation on tasks that require combining events, with benefits dependent on the task and event specification, the largest residual system-error category in our diagnostic analysis.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-15
The authors' main claim, as the classifier picked it out 1.00
“These results support executable aggregation on tasks that require combining events, with benefits dependent on the task and event specification, the largest residual system-error category in our diagnostic analysis.”
How the main claim is stated: asserted 0.78
What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.91 · human participants 0.05 · controlled comparison 0.92 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.97 · mentions code or data availability 0.04
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Questions about an agent's history often require aggregating events, which retrieval alone does not resolve. | background 0.73 | 0.10 | 0.06 | asserted 0.97 |
| We combine source-preserving evidence access with executable aggregation. | contribution 0.87 | 0.87 | 0.10 | asserted 0.98 |
| Trajectory Memory (TM) retains the original trajectory and constructs a source-addressable reading view at question time. | method 0.88 | 0.76 | 0.05 | asserted 1.00 |
| TallyMem augments it with a reusable event ledger, model-generated event specifications and deterministic execution. | method 0.87 | 0.84 | 0.05 | asserted 0.97 |
| With a commit-specific adaptation on 776 questions from 99 held-out MINTEval repositories, TallyMem raises Qwen3-32B exact match from 35.2% to 49.0%, a 13.8-point gain over TM; gains span 12.8-13.9 points across three readers. | result 1.00 | 0.96 | 0.17 | asserted 1.00 |
| A controlled probe with event sets parsed independently from raw logs shows an 11.0-point gain, from 82.8% to 93.8%, without routing. | result 1.00 | 0.94 | 0.06 | asserted 1.00 |
| On full AMA-Bench, the routed system yields a smaller 1.3-point gain, from 69.8% to 71.1%. | result 0.99 | 0.94 | 0.06 | asserted 1.00 |
| These results support executable aggregation on tasks that require combining events, with benefits dependent on the task and event specification, the largest residual system-error category in our diagnostic analysis. | interpretation 1.00 | 0.93 | 0.83 | asserted 0.78 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.96
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-15: 1 request, 6,294 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (12)
Figures are the authors', reproduced from the paper.
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment