Pith.

Artificial Intelligence · arXiv:2610.34767 v1 · submitted 26 Oct 2026 · 49 references

The Right Fact, the Wrong Decision: SciValidityBench for Scientific Agent Memory

Yuexi Pan, Yongchao Chen

The paper says

“Most agent-memory benchmarks frame long-term memory as retrieval: can an agent recover the relevant record from a long and evolving history? Scientific agents face a stricter problem: a correctly remembered fact may still be invalid to use.”

more of the abstract

The same reported test result can support confirmation when the outcome and stopping rule were fixed in advance, yet only exploration when the outcome was selected from twenty inspected results. We call this scientific use validity: whether remembered evidence is eligible to influence a decision under the state governing its interpretation. We introduce SciValidityBench, an executable benchmark factory with 64 TaskSpecs spanning five validity modules and six scientific domains. TaskSpecs couple generated experimental histories with scientific-state definitions, programmatic decision oracles, controlled interventions, and automatic scoring. The SciStateForge compiler-reader framework turns distributed histories into explicit decision-governing state. Counterfactual pairs and targeted corruption-repair controls then test whether models use those dependencies and localize failures. Across four model families, compiled scientific state improves valid decision-making by 9.2-18.8 percentage points over a generic Event Graph. Retrieval alone does not reproduce this cross-family gain, while content-identical flat and typed views show that the improvement comes from exposing governing state rather than a particular graph serialization. SciValidityBench thus shifts memory evaluation from finding the right fact to deciding whether that fact should control the decision.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 27 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up. A clean, well-executed benchmark that formalizes a real gap in agent memory evaluation; the experimental design is thorough and the main result holds. This line is wrong

4 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 49 references say.

4 minor objections

The paper says, at §3.2

Pith says: minor objection 1 of 4

The TaskSpec contract is described functionally but could benefit from a concrete inline example (beyond the Appendix) that traces a single world through history generation, state compilation, and oracle scoring. The current exposition is abstract and may be hard to follow on first reading.

The paper says, at Table 3

Pith says: minor objection 2 of 4

The absolute rates for BM25 are very close to Raw Events across all readers, and the paper states that BM25 does not reproduce the cross-family gain. It would be helpful to add a brief explanation of why BM25 fails to improve even over raw events despite retrieving fewer chunks—is it because the retrieved chunks lack the cross-record dependencies needed for scientific use validity?

show 2 more

The paper says, at §4.4 / Figure 4A

Pith says: minor objection 3 of 4

The module-level gains show negative values for Sunk Cost under Qwen and Gemma (SVL minus Event Graph), yet counterfactual relation accuracy for Sunk Cost is 1.000 for those readers in Table 19. The paper explains that Sunk Cost requires suppressing irrelevant history, but the negative single-world gain versus near-perfect pair performance seems contradictory. Please clarify whether the negative gain reflects a representation-specific failure or a statistical fluctuation due to the small number of Sunk Cost tasks.

The paper says, at §3.3

Pith says: minor objection 4 of 4

The four-pass compilation is described, but the text could more explicitly state what the compiler adds that a careful human annotator could not: it formalizes the deterministic derivation of state from the trace. A brief sentence contrasting the compiler's output with the raw Event Graph's content would reinforce the novelty.

What would settle it

Re-implement the SciStateForge compiler for the Inferential Ledger module following the specification in §3.3 and Appendix B.2. Generate 100 seeded worlds from the provided TaskSpec and compare the oracle actions computed from the re-implemented state against the actions provided in the paper's data. If any oracle action differs, the compiler specification is underspecified or incorrect.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 27 Oct 2026

  1. Reader holds up confidence moderate
  2. Skeptic agrees 28 s · 26,968 tokens read, 2,525 written
  3. Editor wrote the verdict above 12 s · 27,732 tokens read, 1,022 written
  4. Referee minor revision: 4 minor 22 s · 27,257 tokens read, 2,010 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

This paper delivers what it promises: a benchmark that separates “did the agent retrieve the right fact?” from “did the agent know whether that fact is allowed to govern the decision?” That’s a real gap, and the paper formalizes it clearly through five validity modules and an executable TaskSpec factory. The experimental design is the strongest part: counterfactual pairs, corruption-and-repair diagnostics, multiple memory views, and independent replications. The main result—compiled state improves valid decisions by 9.2–18.8 points over a generic event graph—holds across four model families and survives sensitivity checks. I trust the numbers.

The paper is also honest about its boundaries. The compiler’s relations are human-defined and may not exhaust real-world validity concerns; the authors say this explicitly. The Sunk Cost module shows that more structured state can actually hurt, which is a useful reminder that “more compilation” isn’t always better. The weaker readers (Mistral, Phi) show smaller gains and lower counterfactual consistency, but that’s a feature—the benchmark discriminates capability rather than collapsing everything into a single score.

Soft spots: The five modules cover a useful range but aren’t claimed to be exhaustive, and adding new validity mechanisms requires manual rule definition. That’s fine for a first release. The main assumption—that the compiler correctly captures the governing state—is a design choice, not a hidden flaw. Also, the hosted replications use closed models (GPT-5.x) which are less transparent, but the open-weight results are solid and the pattern is consistent.

This is a serious paper with a clear conceptual contribution and reusable infrastructure. It deserves a rigorous peer review and I expect it will be accepted with minor revisions. I’d bring it to my reading group.

The strongest objection

The central claim—that compiled scientific state improves valid decision-making over a generic Event Graph—is supported by a well-controlled experimental design. The benchmark uses counterfactual pairs, corruption–repair diagnostics, multiple memory views, and independent replications to isolate the effect of state compilation. The primary result (9.2–18.8 percentage point gain) holds across four model families, survives removal of exact-choice-overlap tasks, and is consistent with hosted and independent replications. The strongest assumed condition—that the compiler correctly captures the relevant scientific state for each module—is stated explicitly in §3.3 and is a design choice rather than a hidden weakness. The paper acknowledges that the relation derivations are human-defined and may not exhaust real-world validity concerns (§3.2, limitation in §5). Within the benchmark's stated scope, the evidence for the central claim is solid. I find no internal inconsistency, unsupported extrapolation, or methodological flaw that undermines the conclusion as presented.

The paper's central claim is well-supported by a rigorous evaluation. The reader identified the compiler's completeness as the weakest assumption, which is correct but not a critical flaw; the paper's claims are scoped to the defined modules. My stress-test confirms that the experimental design controls for alternative explanations (retrieval, length, presentation, answer bias) and that diagnostic interventions (counterfactual pairs, repair specificity) provide interpretable insights. No hidden assumption or oversight was found. Therefore the verdict should remain ACCEPT.

The referee's report

The paper addresses an important and underappreciated gap in evaluating scientific agent memory: distinguishing the retrieval of a correct fact from the ability to decide whether that fact is eligible to govern an action. The experimental design is comprehensive and rigorous, including multiple control representations (Raw, BM25, Event Graph, Flat SSG, typed SVL), counterfactual pairs, targeted corruption and repair, source-level controls, and independent replications. The central result is consistent across four model families, survives removal of exact-choice-overlap tasks, and is supported by positive 95% confidence intervals throughout. If the result holds, it shifts the focus of memory evaluation from factual recall to the reconstruction of governing state, with clear implications for the design of scientific AI agents. The benchmark is open-source and extensible, which should facilitate further research.

Pith's reading of the paper (a machine's summary, not the authors')

Most agent memory benchmarks ask whether an agent can retrieve a relevant fact from a long history. This paper argues that scientific agents face a stricter problem: a fact that is correctly remembered may still be invalid to use, because its evidential status depends on the scientific state governing its interpretation—such as whether a stopping rule was fixed in advance or an outcome was selected from many. The authors introduce SciValidityBench, an executable benchmark factory with 64 task specifications spanning five validity mechanisms and six scientific domains. The central finding is that compiling distributed experimental histories into explicit decision-governing state improves valid action selection by 9.2–18.8 percentage points over a generic event graph across four model families, and this gain cannot be reproduced by retrieval, longer inputs, or graph structure alone.

What carries the argument. SciStateForge, a compiler–reader pipeline that normalizes domain-specific experimental histories into canonical events, then derives typed relations (e.g., SELECTED FROM, STOPPED AFTER) to produce a Scientific State Graph (SSG) or typed Scientific Validity Ledger (SVL). The compiled state materializes cross-record consequences—such as whether an outcome was pre-registered or adaptively selected—that determine how each recorded fact may be used. This compiled state is what distinguishes the approach from a generic event graph or retrieval.

Core claim. The paper formalizes scientific use validity: whether remembered evidence is eligible to influence a decision under the state governing its interpretation. It operationalizes this via SciValidityBench, where each TaskSpec couples a generated history with a deterministic compiler (SciStateForge) that derives cross-record governing relations such as SELECTED FROM, STOPPED AFTER, and COMPARABLE WITH. Across 640 seeded worlds and four model families, exposing compiled state (flat records or typed ledger) raises valid-action rates by 9.2–18.8 percentage points over a generic Event Graph that preserves all events but omits scientific-state derivation. Counterfactual pairs and targeted corruption–repair controls confirm that the improvement reflects genuine state use rather than answer preference or generic text effects.

The premise it rests on. The SciStateForge compiler's relation derivations (e.g., SELECTED FROM, STOPPED AFTER) are human-defined for each module; if these do not capture the full set of ways scientific evidence becomes invalid, the benchmark may misrepresent an agent's use validity.

If the paper is right (Pith's inferences, not the authors')

Agent memory evaluation should move from retrieval accuracy to state-conditioned decision validity: the same fact can license different actions depending on the governing scientific state.

The 9.2–18.8 percentage point gain indicates that current models cannot reliably reconstruct cross-record dependencies from raw event sequences alone, highlighting a capability gap.

Counterfactual and repair diagnostics reveal that stronger readers (Qwen, Gemma) use compiled state almost perfectly, while weaker readers (Mistral, Phi) show stable answer preferences that produce correct isolated decisions but fail state-driven behavior.

The benchmark's modular design separates validity mechanisms from scientific domains, allowing new mechanisms or domains to be added without redefining the evaluation interface.

The concept of scientific use validity may generalize beyond science to any domain where fact eligibility depends on procedural state, such as legal evidence, medical diagnosis, or financial auditing.

The observed gain from compiled state suggests that future memory architectures should incorporate explicit state tracking, perhaps through differentiable state machines rather than pure retrieval, to resolve cross-record dependencies.

The finding that more history (Event Graph) does not help implies that length-focused retrieval improvements are insufficient for valid decision-making; the bottleneck is state reconstruction, not recall.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Most agent-memory benchmarks frame long-term memory as retrieval: can an agent recover the relevant record from a long and evolving history? Scientific agents face a stricter problem: a correctly remembered fact may still be invalid to use. The same reported test result can support confirmation when the outcome and stopping rule were fixed in advance, yet only exploration when the outcome was selected from twenty inspected results. We call this scientific use validity: whether remembered evidence is eligible to influence a decision under the state governing its interpretation. We introduce SciValidityBench, an executable benchmark factory with 64 TaskSpecs spanning five validity modules and six scientific domains. TaskSpecs couple generated experimental histories with scientific-state definitions, programmatic decision oracles, controlled interventions, and automatic scoring. The SciStateForge compiler-reader framework turns distributed histories into explicit decision-governing state. Counterfactual pairs and targeted corruption-repair controls then test whether models use those dependencies and localize failures. Across four model families, compiled scientific state improves valid decision-making by 9.2-18.8 percentage points over a generic Event Graph. Retrieval alone does not reproduce this cross-family gain, while content-identical flat and typed views show that the improvement comes from exposing governing state rather than a particular graph serialization. SciValidityBench thus shifts memory evaluation from finding the right fact to deciding whether that fact should control the decision.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-27

The authors' main claim, as the classifier picked it out 0.91

“SciValidityBench thus shifts memory evaluation from finding the right fact to deciding whether that fact should control the decision.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.12 · states limitation 0.21 · human participants 0.04 · controlled comparison 0.91 · randomized assignment 0.03 · states quantitative result 0.98 · compares with prior results 0.94 · mentions code or data availability 0.25

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Most agent-memory benchmarks frame long-term memory as retrieval: can an agent recover the relevant record from a long and evolving history? background 1.00 0.05 0.02 asserted 1.00
Scientific agents face a stricter problem: a correctly remembered fact may still be invalid to use. gap 0.68 0.08 0.10 asserted 0.97
The same reported test result can support confirmation when the outcome and stopping rule were fixed in advance, yet only exploration when the outcome was selected from twenty inspected results. background 0.92 0.11 0.05 asserted 0.99
We call this scientific use validity: whether remembered evidence is eligible to influence a decision under the state governing its interpretation. background 0.41 0.33 0.10 asserted 0.97
We introduce SciValidityBench, an executable benchmark factory with 64 TaskSpecs spanning five validity modules and six scientific domains. contribution 0.89 0.94 0.05 asserted 1.00
TaskSpecs couple generated experimental histories with scientific-state definitions, programmatic decision oracles, controlled interventions, and automatic scoring. method 0.94 0.86 0.03 asserted 1.00
The SciStateForge compiler-reader framework turns distributed histories into explicit decision-governing state. method 0.97 0.82 0.04 asserted 0.99
Counterfactual pairs and targeted corruption-repair controls then test whether models use those dependencies and localize failures. method 0.98 0.89 0.04 asserted 0.99
Across four model families, compiled scientific state improves valid decision-making by 9.2-18.8 percentage points over a generic Event Graph. result 1.00 0.95 0.24 asserted 1.00
Retrieval alone does not reproduce this cross-family gain, while content-identical flat and typed views show that the improvement comes from exposing governing state rather than a particular graph serialization. interpretation 0.65 0.95 0.22 asserted 1.00
SciValidityBench thus shifts memory evaluation from finding the right fact to deciding whether that fact should control the decision. interpretation 0.97 0.61 0.87 asserted 1.00

What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from observation or experiment 0.72

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-27: 1 request, 8,123 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (4)
Figure 1 of the paper
Figure 1. The right fact can license different decisions. Retrieval alone cannot distinguish the two worlds because the reported value is identical. SCIVALIDITYBENCH evaluates whether an agent reconstructs the cross-record state that determines how the evidence may be used. This is the distinction between remembering a result and using it validly. arises when an outcome is viewed before a plan is locked, identically named measurements use different constructs, a blank denotes failure rather than zero, or sunk costs influence a prospective choice. Each case links a locally correct record to a…
Figure 2 of the paper
Figure 2. The SCISTATEFORGE pipeline. A domain-specific history is normalized, compiled into decision-governing state, rendered for a model, and scored by an executable oracle. The worked pair changes one scientific-state condition while keeping the task semantics fixed. This step is what distinguishes SCISTATEFORGE from a generic Event Graph. The latter preserves what happened and connects events through sequence or shared objects. The compiler additionally resolves what those occurrences mean for the pending decision. Its flat SSG and typed Scientific Valid￾ity Ledger renderings expose…
Figure 3 of the paper
Figure 3. Main effect and independent replications. Panel A reports the TaskSpec-balanced primary contrast. Panel B shows the same contrast in two hosted seed blocks (H1/H2) and a local Qwen3-14B replication (L). STOPPED AFTER. The reader no longer has to reconstruct those cross-record dependencies from sequence alone. The 9.2–18.8 point gain comes from exposing the scientific relations that govern the current decision, not from organizing the same event history as a generic graph. Flat SSG isolates compiled content from ledger syntax. It reaches 0.991/0.903/0.692/0.609 across the four…
Figure 4 of the paper
Figure 4. State-use and failure diagnosis. Panel A reports module-level validity gains (SVL−Event Graph, percentage points) and shows where compilation helps. Value pairs test whether behavior follows a changed state. Target and matched repairs distinguish restoration of the governing infor￾mation from a generic edit effect. All panels use a shared reader order and Panel A labels every module–reader contrast. 5 CONCLUSION Scientific memory must recover a true record and determine whether it is licensed for the current decision. SCIVALIDITYBENCH makes this scientific-use validity executable…

Figures are the authors', reproduced from the paper.

References (49)
  1. [1] Maharana, Adyasha and Lee, Dong-Ho and Tulyakov, Sergey and Bansal, Mohit and Barbieri, Francesco and Fang, Yuwei. Evaluating Very Long-Term Conversational Memory of LLM Agents. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: ... on Pithdoi:10.18653/v1/2024.acl-long.747
  2. [2] Wu, Di and Wang, Hongwei and Yu, Wenhao and Zhang, Yuwei and Chang, Kai-Wei and Yu, Dong , booktitle = on Pith
  3. [3] Yuanzhe Hu and Yu Wang and Julian McAuley , booktitle=. Evaluating Memory in. 2026 , url= on Pith
  4. [4] Jiayang, Cheng and Ru, Dongyu and Qiu, Lin and Li, Yiyang and Cao, Xuezhi and Song, Yangqiu and Cai, Xunliang , booktitle = on Pith
  5. [5] Brian A. Nosek and Charles R. Ebersole and Alexander C. DeHaven and David T. Mellor , title =. Proceedings of the National Academy of Sciences , volume =. 2018 , doi = on Pith
  6. [6] Optimal Inference After Model Selection , author=. 2017 , eprint= on Pith
  7. [7] Jonathan Bragg and Mike D'Arcy and Nishant Balepur and Dan Bareket and Bhavana Dalvi and Sergey Feldman and Dany Haddad and Jena D. Hwang and Peter Jansen and Varsha Kishore and Bodhisattwa Prasad Majumder and Aakanksha Naik and Sigal Rahamimov and Kyle Richardson and Amanpree... on PithPith's reviewarXiv
  8. [8] Panigrahi, Siba Smarak and Videnovi\'. International Conference on Learning Representations , editor = on Pith
  9. [9] Kon, Patrick Tser Jern and Ding, Qiuyi and Liu, Jiachen and Zhu, Xinyi and Peng, Jingjia and Xing, Jiarong and Huang, Yibo and Qiu, Yiming and Srinivasa, Jayanth and Lee, Myungjin and Chowdhury, Mosharaf and Zaharia, Matei and Chen, Ang , booktitle = on Pith
  10. [10] Jiaqi Shao and Yuxiang Lin and Munish Prasad Lohani and Yufeng Miao and Bing Luo , booktitle=. Do. 2026 , url= on Pith
  11. [11] Park, Joon Sung and O'Brien, Joseph and Cai, Carrie Jun and Morris, Meredith Ringel and Liang, Percy and Bernstein, Michael S. , title =. 2023 , isbn =. doi:10.1145/3586183.3606763 , booktitle = on Pithdoi:10.1145/3586183.3606763
  12. [12] Charles Packer and Sarah Wooders and Kevin Lin and Vivian Fang and Shishir G. Patil and Ion Stoica and Joseph E. Gonzalez , year=. 2310.08560 , archivePrefix= on PithPith's reviewarXiv
  13. [13] Prateek Chhikara and Dev Khant and Saket Aryan and Taranjeet Singh and Deshraj Yadav , year=. 2504.19413 , archivePrefix= on PithPith's reviewarXiv
  14. [14] Zhang, Zeyu and Dai, Quanyu and Bo, Xiaohe and Ma, Chen and Li, Rui and Chen, Xu and Zhu, Jieming and Dong, Zhenhua and Wen, Ji-Rong , title =. 2025 , issue_date =. doi:10.1145/3748302 , journal = on Pithdoi:10.1145/3748302
  15. [15] Tan, Haoran and Zhang, Zeyu and Ma, Chen and Chen, Xu and Dai, Quanyu and Dong, Zhenhua. M em B ench: Towards More Comprehensive Evaluation on the Memory of LLM -based Agents. Findings of the Association for Computational Linguistics: ACL 2025. 2025. doi:10.18653/v1/2025.findi... on Pithdoi:10.18653/v1/2025.findings-acl.989
  16. [16] Tavakoli, Mohammad and Salemi, Alireza and Ye, Carrie and Abdalla, Mohamed and Zamani, Hamed and Mitchell, J Ross , booktitle =. 2026 , url = on Pith
  17. [17] Uddin, Md Nayem and Shubham, Kumar and Blanco, Eduardo and Baral, Chitta and Wang, Gengyu. From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents. Findings of the A ssociation for C omputational L inguistics: ACL 2026. 2026. doi:10.18653/v1/2026.findi... on Pithdoi:10.18653/v1/2026.findings-acl.1337
  18. [18] Yang, Ruozhen and Jiang, Yucheng and Jiang, Yueqi and Kargupta, Priyanka and Zhang, Yunyi and Han, Jiawei. Grounding Agent Memory in Contextual Intent. Findings of the A ssociation for C omputational L inguistics: ACL 2026. 2026. doi:10.18653/v1/2026.findings-acl.584 on Pithdoi:10.18653/v1/2026.findings-acl.584
  19. [19] Shen, Yiting and Li, Kun and Zhou, Wei and Hu, Songlin. M em2 A ct B ench: A Benchmark for Evaluating Long-Term Memory Utilization in Task-Oriented Autonomous Agents. Proceedings of the 64th Annual Meeting of the A ssociation for C omputational L inguistics (Volume 1: Long Pap... on Pithdoi:10.18653/v1/2026.acl-long.370
  20. [20] Zexue He and Yu Wang and Churan Zhi and Yuanzhe Hu and Tzu-Ping Chen and Lang Yin and Ze Chen and Tong Arthur Wu and Siru Ouyang and Zihan Wang and Jiaxin Pei and Julian McAuley and Yejin Choi and Alex Pentland , year=. 2602.16313 , archivePrefix= on PithPith's reviewarXiv
show 29 more
  1. [21] Yujie Zhao and Boqin Yuan and Junbo Huang and Haocheng Yuan and Zhongming Yu and Haozhou Xu and Lanxiang Hu and Abhilash Shankarampeta and Zimeng Huang and Wentao Ni and Yuandong Tian and Jishen Zhao , year=. 2602.22769 , archivePrefix= on PithPith's reviewarXiv
  2. [22] Di Wu and Zixiang Ji and Asmi Kawatkar and Bryan Kwan and Jia-Chen Gu and Nanyun Peng and Kai-Wei Chang , year=. 2605.12493 , archivePrefix= on PithPith's reviewarXiv
  3. [23] Hanxiang Chao and Yihan Bai and Rui Sheng and Tianle Li and Yushi Sun , year=. 2605.06527 , archivePrefix= on PithPith's reviewarXiv
  4. [24] Jiayao Chen and Shi Liu and Linyi Yang , year=. 2606.11851 , archivePrefix= on PithPith's reviewarXiv
  5. [25] Beyond Memory Leaderboards: Evaluating Scientific Memory as Budgeted Context Restoration , author=. 2026 , eprint= on Pith
  6. [26] Ishir Garg and Neel Kolhe and Dawn Song and Xuandong Zhao , year=. 2605.26667 , archivePrefix= on PithPith's reviewarXiv
  7. [27] Xinle Deng and Ruobin Zhong and Hujin Peng and Xiaoben Lu and Yanzhe Wu and Guang Li and Buqiang Xu and Yunzhi Yao and Jizhan Fang and Haoliang Cao and Junjie Guo and Yuan Yuan and Ziqing Ma and Yuanqiang Yu and Rui Hu and Baohua Dong and Hangcheng Zhu and Ningyu Zhang , year=... on PithPith's reviewarXiv
  8. [28] Akash Bonagiri and Devang Borkar and Gerard Janno Anderias and Setareh Rafatirad and Houman Homayoun , year=. 2605.25338 , archivePrefix= on PithPith's reviewarXiv
  9. [29] Jaineet Shah , year=. Causal Agent Replay: Counterfactual Attribution for. 2606.08275 , archivePrefix= on PithPith's reviewarXiv
  10. [30] Wang, Xiaoxuan and Hu, Ziniu and Lu, Pan and Zhu, Yanqiao and Zhang, Jieyu and Subramaniam, Satyen and Loomba, Arjun R and Zhang, Shichang and Sun, Yizhou and Wang, Wei , booktitle =. 2024 , editor = on Pith
  11. [31] Jon M. Laurent and Joseph D. Janizek and Michael Ruzo and Michaela M. Hinks and Michael J. Hammerling and Siddharth Narayanan and Manvitha Ponnapati and Andrew D. White and Samuel G. Rodriques , year=. 2407.10362 , archivePrefix= on PithPith's reviewarXiv
  12. [32] Majumder, Bodhisattwa Prasad and Surana, Harshit and Agarwal, Dhruv and Dalvi Mishra, Bhavana and Meena, Abhijeetsingh and Prakhar, Aryan and Vora, Tirth and Khot, Tushar and Sabharwal, Ashish and Clark, Peter , booktitle = on Pith
  13. [33] Gu, Ken and Shang, Ruoxi and Jiang, Ruien and Kuang, Keying and Lin, Richard-John and Lyu, Donghe and Mao, Yue and Pan, Youran and Wu, Teng and Yu, Jiaqian and Zhang, Yikun and Zhang, Tianmai M. and Zhu, Lanyi and Merrill, Mike A and Heer, Jeffrey and Althoff, Tim. BLADE : Ben... on Pithdoi:10.18653/v1/2024.findings-emnlp.815
  14. [34] Chen, Ziru and Chen, Shijie and Ning, Yuting and Zhang, Qianheng and Wang, Boshi and Yu, Botao and Li, Yifei and Liao, Zeyi and Wei, Chen and Lu, Zitong and Dey, Vishal and Xue, Mingyi and Baker, Frazier N. and Burns, Benjamin and Adu-Ampratwum, Daniel and Huang, Xuhui and Nin... on Pith
  15. [35] Moreau, Luc and Missier, Paolo , title =. 2013 , month = apr, howpublished = on Pith
  16. [36] Sayash Kapoor and Arvind Narayanan , keywords =. Leakage and the reproducibility crisis in machine-learning-based science , journal =. 2023 , issn =. doi:10.1016/j.patter.2023.100804 , url = on Pithdoi:10.1016/j.patter.2023.100804
  17. [37] Ribeiro, Marco Tulio and Wu, Tongshuang and Guestrin, Carlos and Singh, Sameer. Beyond Accuracy: Behavioral Testing of NLP Models with C heck L ist. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020. doi:10.18653/v1/2020.acl-main.442 on Pithdoi:10.18653/v1/2020.acl-main.442
  18. [38] Gardner, Matt and Artzi, Yoav and Basmov, Victoria and Berant, Jonathan and Bogin, Ben and Chen, Sihao and Dasigi, Pradeep and Dua, Dheeru and Elazar, Yanai and Gottumukkala, Ananth and Gupta, Nitish and Hajishirzi, Hannaneh and Ilharco, Gabriel and Khashabi, Daniel and Lin, K... on Pith
  19. [39] Explicit State Elicitation Is Not Enough: A Controlled Audit of Memory-Policy Classification , author=. 2026 , eprint= on Pith
  20. [40] Yuyao Wang and Zhongjian Zhang and Mo Chi and Kaichi Yu and Yuhan Li and Miao Peng and Bing Tong and Chen Zhang and Yan Zhou and Jia Li , year=. 2605.18421 , archivePrefix= on PithPith's reviewarXiv
  21. [41] Benchmarking Continual Agent Memory for Online Learning, Transfer, and Forgetting , author=. ICLR 2026 Workshop on Lifelong Agents: Learning, Aligning, Evolving , year= on Pith
  22. [42] Qingyao Ai and Yichen Tang and Changyue Wang and Jianming Long and Weihang Su and Yiqun LIU , booktitle=. 2026 , url= on Pith
  23. [43] Enze Ma and Yufan Zhou and Wei-Chieh Huang and Jie Yang and Huanhuan Ma and Zixuan Wang and Chengze Li and Chunyu Miao and Philip S. Yu and Zhen Wang , year=. 2606.24595 , archivePrefix= on PithPith's reviewarXiv
  24. [44] Wang, Xinming and Xu, Jian and Lian, Sheng and Chen, Yi and Guo, Haiyang and Zhu, Fei and Shao, Yuanqi and Ren, Minsi and Yi, Hongzhu and Feng, Aslan H. and Yang, Hongming and Wu, Tailin and Hu, Han and Xiang, Shiming and Zhang, Xu-Yao and Liu, Cheng-Lin , journal=. The Hitchh... on Pithdoi:10.1109/tpami.2026.3721500
  25. [45] Qwen3 Technical Report , author=. 2025 , eprint= on Pith
  26. [46] Gemma 3 Technical Report , author=. 2025 , eprint= on Pith
  27. [47] Albert Q. Jiang and Alexandre Sablayrolles and Arthur Mensch and Chris Bamford and Devendra Singh Chaplot and Diego de las Casas and Florian Bressand and Gianna Lengyel and Guillaume Lample and Lucile Saulnier and Lélio Renard Lavaud and Marie-Anne Lachaux and Pierre Stock and... on PithPith's reviewarXiv
  28. [48] Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone , author=. 2024 , eprint= on Pith
  29. [49] Kwon, Woosuk and Li, Zhuohan and Zhuang, Siyuan and Sheng, Ying and Zheng, Lianmin and Yu, Cody Hao and Gonzalez, Joseph and Zhang, Hao and Stoica, Ion , title =. 2023 , isbn =. doi:10.1145/3600006.3613165 , booktitle = on Pithdoi:10.1145/3600006.3613165

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment