Artificial Intelligence · arXiv:2610.34767 v1 · submitted 26 Oct 2026 · 49 references
The Right Fact, the Wrong Decision: SciValidityBench for Scientific Agent Memory
The paper says
“Most agent-memory benchmarks frame long-term memory as retrieval: can an agent recover the relevant record from a long and evolving history? Scientific agents face a stricter problem: a correctly remembered fact may still be invalid to use.”
more of the abstract
The same reported test result can support confirmation when the outcome and stopping rule were fixed in advance, yet only exploration when the outcome was selected from twenty inspected results. We call this scientific use validity: whether remembered evidence is eligible to influence a decision under the state governing its interpretation. We introduce SciValidityBench, an executable benchmark factory with 64 TaskSpecs spanning five validity modules and six scientific domains. TaskSpecs couple generated experimental histories with scientific-state definitions, programmatic decision oracles, controlled interventions, and automatic scoring. The SciStateForge compiler-reader framework turns distributed histories into explicit decision-governing state. Counterfactual pairs and targeted corruption-repair controls then test whether models use those dependencies and localize failures. Across four model families, compiled scientific state improves valid decision-making by 9.2-18.8 percentage points over a generic Event Graph. Retrieval alone does not reproduce this cross-family gain, while content-identical flat and typed views show that the improvement comes from exposing governing state rather than a particular graph serialization. SciValidityBench thus shifts memory evaluation from finding the right fact to deciding whether that fact should control the decision.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 27 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up. A clean, well-executed benchmark that formalizes a real gap in agent memory evaluation; the experimental design is thorough and the main result holds. This line is wrong
4 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 49 references say.
4 minor objections
The paper says, at §3.2
Pith says: minor objection 1 of 4
The TaskSpec contract is described functionally but could benefit from a concrete inline example (beyond the Appendix) that traces a single world through history generation, state compilation, and oracle scoring. The current exposition is abstract and may be hard to follow on first reading.
The paper says, at Table 3
Pith says: minor objection 2 of 4
The absolute rates for BM25 are very close to Raw Events across all readers, and the paper states that BM25 does not reproduce the cross-family gain. It would be helpful to add a brief explanation of why BM25 fails to improve even over raw events despite retrieving fewer chunks—is it because the retrieved chunks lack the cross-record dependencies needed for scientific use validity?
show 2 more
The paper says, at §4.4 / Figure 4A
Pith says: minor objection 3 of 4
The module-level gains show negative values for Sunk Cost under Qwen and Gemma (SVL minus Event Graph), yet counterfactual relation accuracy for Sunk Cost is 1.000 for those readers in Table 19. The paper explains that Sunk Cost requires suppressing irrelevant history, but the negative single-world gain versus near-perfect pair performance seems contradictory. Please clarify whether the negative gain reflects a representation-specific failure or a statistical fluctuation due to the small number of Sunk Cost tasks.
The paper says, at §3.3
Pith says: minor objection 4 of 4
The four-pass compilation is described, but the text could more explicitly state what the compiler adds that a careful human annotator could not: it formalizes the deterministic derivation of state from the trace. A brief sentence contrasting the compiler's output with the raw Event Graph's content would reinforce the novelty.
What would settle it
Re-implement the SciStateForge compiler for the Inferential Ledger module following the specification in §3.3 and Appendix B.2. Generate 100 seeded worlds from the provided TaskSpec and compare the oracle actions computed from the re-implemented state against the actions provided in the paper's data. If any oracle action differs, the compiler specification is underspecified or incorrect.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 27 Oct 2026
- Reader holds up
- Skeptic agrees
- Editor wrote the verdict above
- Referee minor revision: 4 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
This paper delivers what it promises: a benchmark that separates “did the agent retrieve the right fact?” from “did the agent know whether that fact is allowed to govern the decision?” That’s a real gap, and the paper formalizes it clearly through five validity modules and an executable TaskSpec factory. The experimental design is the strongest part: counterfactual pairs, corruption-and-repair diagnostics, multiple memory views, and independent replications. The main result—compiled state improves valid decisions by 9.2–18.8 points over a generic event graph—holds across four model families and survives sensitivity checks. I trust the numbers.
The paper is also honest about its boundaries. The compiler’s relations are human-defined and may not exhaust real-world validity concerns; the authors say this explicitly. The Sunk Cost module shows that more structured state can actually hurt, which is a useful reminder that “more compilation” isn’t always better. The weaker readers (Mistral, Phi) show smaller gains and lower counterfactual consistency, but that’s a feature—the benchmark discriminates capability rather than collapsing everything into a single score.
Soft spots: The five modules cover a useful range but aren’t claimed to be exhaustive, and adding new validity mechanisms requires manual rule definition. That’s fine for a first release. The main assumption—that the compiler correctly captures the governing state—is a design choice, not a hidden flaw. Also, the hosted replications use closed models (GPT-5.x) which are less transparent, but the open-weight results are solid and the pattern is consistent.
This is a serious paper with a clear conceptual contribution and reusable infrastructure. It deserves a rigorous peer review and I expect it will be accepted with minor revisions. I’d bring it to my reading group.
The strongest objection
The central claim—that compiled scientific state improves valid decision-making over a generic Event Graph—is supported by a well-controlled experimental design. The benchmark uses counterfactual pairs, corruption–repair diagnostics, multiple memory views, and independent replications to isolate the effect of state compilation. The primary result (9.2–18.8 percentage point gain) holds across four model families, survives removal of exact-choice-overlap tasks, and is consistent with hosted and independent replications. The strongest assumed condition—that the compiler correctly captures the relevant scientific state for each module—is stated explicitly in §3.3 and is a design choice rather than a hidden weakness. The paper acknowledges that the relation derivations are human-defined and may not exhaust real-world validity concerns (§3.2, limitation in §5). Within the benchmark's stated scope, the evidence for the central claim is solid. I find no internal inconsistency, unsupported extrapolation, or methodological flaw that undermines the conclusion as presented.
The paper's central claim is well-supported by a rigorous evaluation. The reader identified the compiler's completeness as the weakest assumption, which is correct but not a critical flaw; the paper's claims are scoped to the defined modules. My stress-test confirms that the experimental design controls for alternative explanations (retrieval, length, presentation, answer bias) and that diagnostic interventions (counterfactual pairs, repair specificity) provide interpretable insights. No hidden assumption or oversight was found. Therefore the verdict should remain ACCEPT.
The referee's report
The paper addresses an important and underappreciated gap in evaluating scientific agent memory: distinguishing the retrieval of a correct fact from the ability to decide whether that fact is eligible to govern an action. The experimental design is comprehensive and rigorous, including multiple control representations (Raw, BM25, Event Graph, Flat SSG, typed SVL), counterfactual pairs, targeted corruption and repair, source-level controls, and independent replications. The central result is consistent across four model families, survives removal of exact-choice-overlap tasks, and is supported by positive 95% confidence intervals throughout. If the result holds, it shifts the focus of memory evaluation from factual recall to the reconstruction of governing state, with clear implications for the design of scientific AI agents. The benchmark is open-source and extensible, which should facilitate further research.
Pith's reading of the paper (a machine's summary, not the authors')
Most agent memory benchmarks ask whether an agent can retrieve a relevant fact from a long history. This paper argues that scientific agents face a stricter problem: a fact that is correctly remembered may still be invalid to use, because its evidential status depends on the scientific state governing its interpretation—such as whether a stopping rule was fixed in advance or an outcome was selected from many. The authors introduce SciValidityBench, an executable benchmark factory with 64 task specifications spanning five validity mechanisms and six scientific domains. The central finding is that compiling distributed experimental histories into explicit decision-governing state improves valid action selection by 9.2–18.8 percentage points over a generic event graph across four model families, and this gain cannot be reproduced by retrieval, longer inputs, or graph structure alone.
What carries the argument. SciStateForge, a compiler–reader pipeline that normalizes domain-specific experimental histories into canonical events, then derives typed relations (e.g., SELECTED FROM, STOPPED AFTER) to produce a Scientific State Graph (SSG) or typed Scientific Validity Ledger (SVL). The compiled state materializes cross-record consequences—such as whether an outcome was pre-registered or adaptively selected—that determine how each recorded fact may be used. This compiled state is what distinguishes the approach from a generic event graph or retrieval.
Core claim. The paper formalizes scientific use validity: whether remembered evidence is eligible to influence a decision under the state governing its interpretation. It operationalizes this via SciValidityBench, where each TaskSpec couples a generated history with a deterministic compiler (SciStateForge) that derives cross-record governing relations such as SELECTED FROM, STOPPED AFTER, and COMPARABLE WITH. Across 640 seeded worlds and four model families, exposing compiled state (flat records or typed ledger) raises valid-action rates by 9.2–18.8 percentage points over a generic Event Graph that preserves all events but omits scientific-state derivation. Counterfactual pairs and targeted corruption–repair controls confirm that the improvement reflects genuine state use rather than answer preference or generic text effects.
The premise it rests on. The SciStateForge compiler's relation derivations (e.g., SELECTED FROM, STOPPED AFTER) are human-defined for each module; if these do not capture the full set of ways scientific evidence becomes invalid, the benchmark may misrepresent an agent's use validity.
If the paper is right (Pith's inferences, not the authors')
Agent memory evaluation should move from retrieval accuracy to state-conditioned decision validity: the same fact can license different actions depending on the governing scientific state.
The 9.2–18.8 percentage point gain indicates that current models cannot reliably reconstruct cross-record dependencies from raw event sequences alone, highlighting a capability gap.
Counterfactual and repair diagnostics reveal that stronger readers (Qwen, Gemma) use compiled state almost perfectly, while weaker readers (Mistral, Phi) show stable answer preferences that produce correct isolated decisions but fail state-driven behavior.
The benchmark's modular design separates validity mechanisms from scientific domains, allowing new mechanisms or domains to be added without redefining the evaluation interface.
The concept of scientific use validity may generalize beyond science to any domain where fact eligibility depends on procedural state, such as legal evidence, medical diagnosis, or financial auditing.
The observed gain from compiled state suggests that future memory architectures should incorporate explicit state tracking, perhaps through differentiable state machines rather than pure retrieval, to resolve cross-record dependencies.
The finding that more history (Event Graph) does not help implies that length-focused retrieval improvements are insufficient for valid decision-making; the bottleneck is state reconstruction, not recall.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Most agent-memory benchmarks frame long-term memory as retrieval: can an agent recover the relevant record from a long and evolving history? Scientific agents face a stricter problem: a correctly remembered fact may still be invalid to use. The same reported test result can support confirmation when the outcome and stopping rule were fixed in advance, yet only exploration when the outcome was selected from twenty inspected results. We call this scientific use validity: whether remembered evidence is eligible to influence a decision under the state governing its interpretation. We introduce SciValidityBench, an executable benchmark factory with 64 TaskSpecs spanning five validity modules and six scientific domains. TaskSpecs couple generated experimental histories with scientific-state definitions, programmatic decision oracles, controlled interventions, and automatic scoring. The SciStateForge compiler-reader framework turns distributed histories into explicit decision-governing state. Counterfactual pairs and targeted corruption-repair controls then test whether models use those dependencies and localize failures. Across four model families, compiled scientific state improves valid decision-making by 9.2-18.8 percentage points over a generic Event Graph. Retrieval alone does not reproduce this cross-family gain, while content-identical flat and typed views show that the improvement comes from exposing governing state rather than a particular graph serialization. SciValidityBench thus shifts memory evaluation from finding the right fact to deciding whether that fact should control the decision.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-27
The authors' main claim, as the classifier picked it out 0.91
“SciValidityBench thus shifts memory evaluation from finding the right fact to deciding whether that fact should control the decision.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.12 · states limitation 0.21 · human participants 0.04 · controlled comparison 0.91 · randomized assignment 0.03 · states quantitative result 0.98 · compares with prior results 0.94 · mentions code or data availability 0.25
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Most agent-memory benchmarks frame long-term memory as retrieval: can an agent recover the relevant record from a long and evolving history? | background 1.00 | 0.05 | 0.02 | asserted 1.00 |
| Scientific agents face a stricter problem: a correctly remembered fact may still be invalid to use. | gap 0.68 | 0.08 | 0.10 | asserted 0.97 |
| The same reported test result can support confirmation when the outcome and stopping rule were fixed in advance, yet only exploration when the outcome was selected from twenty inspected results. | background 0.92 | 0.11 | 0.05 | asserted 0.99 |
| We call this scientific use validity: whether remembered evidence is eligible to influence a decision under the state governing its interpretation. | background 0.41 | 0.33 | 0.10 | asserted 0.97 |
| We introduce SciValidityBench, an executable benchmark factory with 64 TaskSpecs spanning five validity modules and six scientific domains. | contribution 0.89 | 0.94 | 0.05 | asserted 1.00 |
| TaskSpecs couple generated experimental histories with scientific-state definitions, programmatic decision oracles, controlled interventions, and automatic scoring. | method 0.94 | 0.86 | 0.03 | asserted 1.00 |
| The SciStateForge compiler-reader framework turns distributed histories into explicit decision-governing state. | method 0.97 | 0.82 | 0.04 | asserted 0.99 |
| Counterfactual pairs and targeted corruption-repair controls then test whether models use those dependencies and localize failures. | method 0.98 | 0.89 | 0.04 | asserted 0.99 |
| Across four model families, compiled scientific state improves valid decision-making by 9.2-18.8 percentage points over a generic Event Graph. | result 1.00 | 0.95 | 0.24 | asserted 1.00 |
| Retrieval alone does not reproduce this cross-family gain, while content-identical flat and typed views show that the improvement comes from exposing governing state rather than a particular graph serialization. | interpretation 0.65 | 0.95 | 0.22 | asserted 1.00 |
| SciValidityBench thus shifts memory evaluation from finding the right fact to deciding whether that fact should control the decision. | interpretation 0.97 | 0.61 | 0.87 | asserted 1.00 |
What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from observation or experiment 0.72
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-27: 1 request, 8,123 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (4)
Figures are the authors', reproduced from the paper.
References (49)
- [1] Maharana, Adyasha and Lee, Dong-Ho and Tulyakov, Sergey and Bansal, Mohit and Barbieri, Francesco and Fang, Yuwei. Evaluating Very Long-Term Conversational Memory of LLM Agents. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: ... on Pithdoi:10.18653/v1/2024.acl-long.747
- [2] Wu, Di and Wang, Hongwei and Yu, Wenhao and Zhang, Yuwei and Chang, Kai-Wei and Yu, Dong , booktitle = on Pith
- [3] Yuanzhe Hu and Yu Wang and Julian McAuley , booktitle=. Evaluating Memory in. 2026 , url= on Pith
- [4] Jiayang, Cheng and Ru, Dongyu and Qiu, Lin and Li, Yiyang and Cao, Xuezhi and Song, Yangqiu and Cai, Xunliang , booktitle = on Pith
- [5] Brian A. Nosek and Charles R. Ebersole and Alexander C. DeHaven and David T. Mellor , title =. Proceedings of the National Academy of Sciences , volume =. 2018 , doi = on Pith
- [6] Optimal Inference After Model Selection , author=. 2017 , eprint= on Pith
- [7] Jonathan Bragg and Mike D'Arcy and Nishant Balepur and Dan Bareket and Bhavana Dalvi and Sergey Feldman and Dany Haddad and Jena D. Hwang and Peter Jansen and Varsha Kishore and Bodhisattwa Prasad Majumder and Aakanksha Naik and Sigal Rahamimov and Kyle Richardson and Amanpree... on PithPith's reviewarXiv
- [8] Panigrahi, Siba Smarak and Videnovi\'. International Conference on Learning Representations , editor = on Pith
- [9] Kon, Patrick Tser Jern and Ding, Qiuyi and Liu, Jiachen and Zhu, Xinyi and Peng, Jingjia and Xing, Jiarong and Huang, Yibo and Qiu, Yiming and Srinivasa, Jayanth and Lee, Myungjin and Chowdhury, Mosharaf and Zaharia, Matei and Chen, Ang , booktitle = on Pith
- [10] Jiaqi Shao and Yuxiang Lin and Munish Prasad Lohani and Yufeng Miao and Bing Luo , booktitle=. Do. 2026 , url= on Pith
- [11] Park, Joon Sung and O'Brien, Joseph and Cai, Carrie Jun and Morris, Meredith Ringel and Liang, Percy and Bernstein, Michael S. , title =. 2023 , isbn =. doi:10.1145/3586183.3606763 , booktitle = on Pithdoi:10.1145/3586183.3606763
- [12] Charles Packer and Sarah Wooders and Kevin Lin and Vivian Fang and Shishir G. Patil and Ion Stoica and Joseph E. Gonzalez , year=. 2310.08560 , archivePrefix= on PithPith's reviewarXiv
- [13] Prateek Chhikara and Dev Khant and Saket Aryan and Taranjeet Singh and Deshraj Yadav , year=. 2504.19413 , archivePrefix= on PithPith's reviewarXiv
- [14] Zhang, Zeyu and Dai, Quanyu and Bo, Xiaohe and Ma, Chen and Li, Rui and Chen, Xu and Zhu, Jieming and Dong, Zhenhua and Wen, Ji-Rong , title =. 2025 , issue_date =. doi:10.1145/3748302 , journal = on Pithdoi:10.1145/3748302
- [15] Tan, Haoran and Zhang, Zeyu and Ma, Chen and Chen, Xu and Dai, Quanyu and Dong, Zhenhua. M em B ench: Towards More Comprehensive Evaluation on the Memory of LLM -based Agents. Findings of the Association for Computational Linguistics: ACL 2025. 2025. doi:10.18653/v1/2025.findi... on Pithdoi:10.18653/v1/2025.findings-acl.989
- [16] Tavakoli, Mohammad and Salemi, Alireza and Ye, Carrie and Abdalla, Mohamed and Zamani, Hamed and Mitchell, J Ross , booktitle =. 2026 , url = on Pith
- [17] Uddin, Md Nayem and Shubham, Kumar and Blanco, Eduardo and Baral, Chitta and Wang, Gengyu. From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents. Findings of the A ssociation for C omputational L inguistics: ACL 2026. 2026. doi:10.18653/v1/2026.findi... on Pithdoi:10.18653/v1/2026.findings-acl.1337
- [18] Yang, Ruozhen and Jiang, Yucheng and Jiang, Yueqi and Kargupta, Priyanka and Zhang, Yunyi and Han, Jiawei. Grounding Agent Memory in Contextual Intent. Findings of the A ssociation for C omputational L inguistics: ACL 2026. 2026. doi:10.18653/v1/2026.findings-acl.584 on Pithdoi:10.18653/v1/2026.findings-acl.584
- [19] Shen, Yiting and Li, Kun and Zhou, Wei and Hu, Songlin. M em2 A ct B ench: A Benchmark for Evaluating Long-Term Memory Utilization in Task-Oriented Autonomous Agents. Proceedings of the 64th Annual Meeting of the A ssociation for C omputational L inguistics (Volume 1: Long Pap... on Pithdoi:10.18653/v1/2026.acl-long.370
- [20] Zexue He and Yu Wang and Churan Zhi and Yuanzhe Hu and Tzu-Ping Chen and Lang Yin and Ze Chen and Tong Arthur Wu and Siru Ouyang and Zihan Wang and Jiaxin Pei and Julian McAuley and Yejin Choi and Alex Pentland , year=. 2602.16313 , archivePrefix= on PithPith's reviewarXiv
show 29 more
- [21] Yujie Zhao and Boqin Yuan and Junbo Huang and Haocheng Yuan and Zhongming Yu and Haozhou Xu and Lanxiang Hu and Abhilash Shankarampeta and Zimeng Huang and Wentao Ni and Yuandong Tian and Jishen Zhao , year=. 2602.22769 , archivePrefix= on PithPith's reviewarXiv
- [22] Di Wu and Zixiang Ji and Asmi Kawatkar and Bryan Kwan and Jia-Chen Gu and Nanyun Peng and Kai-Wei Chang , year=. 2605.12493 , archivePrefix= on PithPith's reviewarXiv
- [23] Hanxiang Chao and Yihan Bai and Rui Sheng and Tianle Li and Yushi Sun , year=. 2605.06527 , archivePrefix= on PithPith's reviewarXiv
- [24] Jiayao Chen and Shi Liu and Linyi Yang , year=. 2606.11851 , archivePrefix= on PithPith's reviewarXiv
- [25] Beyond Memory Leaderboards: Evaluating Scientific Memory as Budgeted Context Restoration , author=. 2026 , eprint= on Pith
- [26] Ishir Garg and Neel Kolhe and Dawn Song and Xuandong Zhao , year=. 2605.26667 , archivePrefix= on PithPith's reviewarXiv
- [27] Xinle Deng and Ruobin Zhong and Hujin Peng and Xiaoben Lu and Yanzhe Wu and Guang Li and Buqiang Xu and Yunzhi Yao and Jizhan Fang and Haoliang Cao and Junjie Guo and Yuan Yuan and Ziqing Ma and Yuanqiang Yu and Rui Hu and Baohua Dong and Hangcheng Zhu and Ningyu Zhang , year=... on PithPith's reviewarXiv
- [28] Akash Bonagiri and Devang Borkar and Gerard Janno Anderias and Setareh Rafatirad and Houman Homayoun , year=. 2605.25338 , archivePrefix= on PithPith's reviewarXiv
- [29] Jaineet Shah , year=. Causal Agent Replay: Counterfactual Attribution for. 2606.08275 , archivePrefix= on PithPith's reviewarXiv
- [30] Wang, Xiaoxuan and Hu, Ziniu and Lu, Pan and Zhu, Yanqiao and Zhang, Jieyu and Subramaniam, Satyen and Loomba, Arjun R and Zhang, Shichang and Sun, Yizhou and Wang, Wei , booktitle =. 2024 , editor = on Pith
- [31] Jon M. Laurent and Joseph D. Janizek and Michael Ruzo and Michaela M. Hinks and Michael J. Hammerling and Siddharth Narayanan and Manvitha Ponnapati and Andrew D. White and Samuel G. Rodriques , year=. 2407.10362 , archivePrefix= on PithPith's reviewarXiv
- [32] Majumder, Bodhisattwa Prasad and Surana, Harshit and Agarwal, Dhruv and Dalvi Mishra, Bhavana and Meena, Abhijeetsingh and Prakhar, Aryan and Vora, Tirth and Khot, Tushar and Sabharwal, Ashish and Clark, Peter , booktitle = on Pith
- [33] Gu, Ken and Shang, Ruoxi and Jiang, Ruien and Kuang, Keying and Lin, Richard-John and Lyu, Donghe and Mao, Yue and Pan, Youran and Wu, Teng and Yu, Jiaqian and Zhang, Yikun and Zhang, Tianmai M. and Zhu, Lanyi and Merrill, Mike A and Heer, Jeffrey and Althoff, Tim. BLADE : Ben... on Pithdoi:10.18653/v1/2024.findings-emnlp.815
- [34] Chen, Ziru and Chen, Shijie and Ning, Yuting and Zhang, Qianheng and Wang, Boshi and Yu, Botao and Li, Yifei and Liao, Zeyi and Wei, Chen and Lu, Zitong and Dey, Vishal and Xue, Mingyi and Baker, Frazier N. and Burns, Benjamin and Adu-Ampratwum, Daniel and Huang, Xuhui and Nin... on Pith
- [35] Moreau, Luc and Missier, Paolo , title =. 2013 , month = apr, howpublished = on Pith
- [36] Sayash Kapoor and Arvind Narayanan , keywords =. Leakage and the reproducibility crisis in machine-learning-based science , journal =. 2023 , issn =. doi:10.1016/j.patter.2023.100804 , url = on Pithdoi:10.1016/j.patter.2023.100804
- [37] Ribeiro, Marco Tulio and Wu, Tongshuang and Guestrin, Carlos and Singh, Sameer. Beyond Accuracy: Behavioral Testing of NLP Models with C heck L ist. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020. doi:10.18653/v1/2020.acl-main.442 on Pithdoi:10.18653/v1/2020.acl-main.442
- [38] Gardner, Matt and Artzi, Yoav and Basmov, Victoria and Berant, Jonathan and Bogin, Ben and Chen, Sihao and Dasigi, Pradeep and Dua, Dheeru and Elazar, Yanai and Gottumukkala, Ananth and Gupta, Nitish and Hajishirzi, Hannaneh and Ilharco, Gabriel and Khashabi, Daniel and Lin, K... on Pith
- [39] Explicit State Elicitation Is Not Enough: A Controlled Audit of Memory-Policy Classification , author=. 2026 , eprint= on Pith
- [40] Yuyao Wang and Zhongjian Zhang and Mo Chi and Kaichi Yu and Yuhan Li and Miao Peng and Bing Tong and Chen Zhang and Yan Zhou and Jia Li , year=. 2605.18421 , archivePrefix= on PithPith's reviewarXiv
- [41] Benchmarking Continual Agent Memory for Online Learning, Transfer, and Forgetting , author=. ICLR 2026 Workshop on Lifelong Agents: Learning, Aligning, Evolving , year= on Pith
- [42] Qingyao Ai and Yichen Tang and Changyue Wang and Jianming Long and Weihang Su and Yiqun LIU , booktitle=. 2026 , url= on Pith
- [43] Enze Ma and Yufan Zhou and Wei-Chieh Huang and Jie Yang and Huanhuan Ma and Zixuan Wang and Chengze Li and Chunyu Miao and Philip S. Yu and Zhen Wang , year=. 2606.24595 , archivePrefix= on PithPith's reviewarXiv
- [44] Wang, Xinming and Xu, Jian and Lian, Sheng and Chen, Yi and Guo, Haiyang and Zhu, Fei and Shao, Yuanqi and Ren, Minsi and Yi, Hongzhu and Feng, Aslan H. and Yang, Hongming and Wu, Tailin and Hu, Han and Xiang, Shiming and Zhang, Xu-Yao and Liu, Cheng-Lin , journal=. The Hitchh... on Pithdoi:10.1109/tpami.2026.3721500
- [45] Qwen3 Technical Report , author=. 2025 , eprint= on Pith
- [46] Gemma 3 Technical Report , author=. 2025 , eprint= on Pith
- [47] Albert Q. Jiang and Alexandre Sablayrolles and Arthur Mensch and Chris Bamford and Devendra Singh Chaplot and Diego de las Casas and Florian Bressand and Gianna Lengyel and Guillaume Lample and Lucile Saulnier and Lélio Renard Lavaud and Marie-Anne Lachaux and Pierre Stock and... on PithPith's reviewarXiv
- [48] Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone , author=. 2024 , eprint= on Pith
- [49] Kwon, Woosuk and Li, Zhuohan and Zhuang, Siyuan and Sheng, Ying and Zheng, Lianmin and Yu, Cody Hao and Gonzalez, Joseph and Zhang, Hao and Stoica, Ion , title =. 2023 , isbn =. doi:10.1145/3600006.3613165 , booktitle = on Pithdoi:10.1145/3600006.3613165
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment