{"paper":{"title":"MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks","arxiv_id":"2602.16313","primary_cat":"cs.CL","submitted_at":"2026-02-18T09:49:14+00:00","version_compiled":"v2"},"notice":"Every quoted sentence is the authors' text from the arXiv listing abstract. Labels and probabilities are the outputs of classifier-1.13.0 on the question bank abstract-p1-20260923, exactly as returned. No sentence in this record was written by a language model.","schema":"pith.compiled-record/v0","status":"compiled","abstract":{"source":"arXiv listing abstract v2","sentences":[{"i":0,"p":0.78,"id":"s0","own":false,"role":"background","text":"Existing evaluations of agents with memory typically assess memorization and action in isolation.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.04,"main_claim_p":0.03,"quantitative":false,"quantitative_p":0.13,"role_probabilities":{"gap":0.22,"method":0.0,"result":0.0,"background":0.78,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":1,"p":0.53,"id":"s1","own":false,"role":"background","text":"One class of benchmarks evaluates memorization by testing recall of past conversations or text but fails to capture how memory is used to guide future decisions.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.04,"main_claim_p":0.03,"quantitative":false,"quantitative_p":0.16,"role_probabilities":{"gap":0.46,"method":0.0,"result":0.0,"background":0.53,"limitation":0.01,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":2,"p":0.81,"id":"s2","own":false,"role":"background","text":"Another class focuses on agents acting in single-session tasks without the need for long-term memory.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.03,"main_claim_p":0.02,"quantitative":false,"quantitative_p":0.09,"role_probabilities":{"gap":0.19,"method":0.0,"result":0.0,"background":0.81,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":3,"p":0.56,"id":"s3","own":false,"role":"background","text":"However, in realistic settings, memorization and action are tightly coupled: agents acquire memory while interacting with the environment, and subsequently rely on that memory to solve future tasks.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.07,"main_claim_p":0.07,"quantitative":false,"quantitative_p":0.05,"role_probabilities":{"gap":0.44,"method":0.0,"result":0.0,"background":0.56,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":4,"p":0.99,"id":"s4","own":true,"role":"contribution","text":"To capture this setting, we introduce MemoryArena, a unified evaluation gym for benchmarking agent memory in multi-session Memory-Agent-Environment loops.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.92,"main_claim_p":0.08,"quantitative":false,"quantitative_p":0.04,"role_probabilities":{"gap":0.0,"method":0.01,"result":0.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.99,"interpretation":0.0}},{"i":5,"p":0.8,"id":"s5","own":true,"role":"method","text":"The benchmark consists of human-crafted agentic tasks with explicitly interdependent subtasks, where agents must learn from earlier actions and feedback by distilling experiences into memory, and subsequently use that memory to guide later actions to solve the overall task.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.8,"main_claim_p":0.11,"quantitative":false,"quantitative_p":0.04,"role_probabilities":{"gap":0.0,"method":0.8,"result":0.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.2,"interpretation":0.0}},{"i":6,"p":0.94,"id":"s6","own":true,"role":"result","text":"MemoryArena supports evaluation across web navigation, preference-constrained planning, progressive information search, and sequential formal reasoning, and reveals that agents with near-saturated performance on existing long-context memory benchmarks like LoCoMo perform poorly in our agentic setting, exposing a gap in current evaluations for agents with memory.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.94,"main_claim_p":0.79,"quantitative":true,"quantitative_p":0.78,"role_probabilities":{"gap":0.0,"method":0.0,"result":0.94,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.02,"interpretation":0.04}},{"i":7,"p":0.91,"id":"s7","own":false,"role":"contribution","text":"MemoryArena is now released at https://memoryarena.github.io/.","hedge":{"p":0.99,"label":"asserted"},"own_p":0.44,"main_claim_p":0.02,"quantitative":false,"quantitative_p":0.03,"role_probabilities":{"gap":0.0,"method":0.02,"result":0.0,"background":0.0,"limitation":0.0,"future_work":0.07,"contribution":0.91,"interpretation":0.0}}]},"compiled":{"main_claim":{"p":1.0,"text":"MemoryArena supports evaluation across web navigation, preference-constrained planning, progressive information search, and sequential formal reasoning, and reveals that agents with near-saturated performance on existing long-context memory benchmarks like LoCoMo perform poorly in our agentic setting, exposing a gap in current evaluations for agents with memory.","location":"abstract s6","hedge_strength":{"p":1.0,"label":"asserted"}},"provenance":{"route":{"via":"TypeSafe API","model":"classifier-1.13.0","endpoint":"https://api.typesafe.ai/v1/systemone","provider":"TypeSafe"},"packing":"one request per paper","requests":1,"input_tokens":6302,"cost_usd_reported":0.0},"compiled_at":"2026-09-23","bank_version":"abstract-p1-20260923","model_version":"classifier-1.13.0","abstract_states":{"claims_first":0.15,"states_limitation":0.14,"human_participants":0.29,"controlled_comparison":0.34,"randomized_assignment":0.01,"states_quantitative_result":0.06,"compares_with_prior_results":0.93,"mentions_code_or_data_availability":0.95},"evidential_basis":{"p":0.94,"value":"observation or experiment","probabilities":{"argument":0.02,"literature":0.0,"proof_or_derivation":0.0,"analysis_of_existing_data":0.0,"observation_or_experiment":0.94,"simulation_or_computation":0.04}},"contribution_type":{"p":1.0,"value":"dataset or benchmark","probabilities":{"review":0.0,"method_or_tool":0.0,"empirical_study":0.0,"simulation_study":0.0,"theoretical_result":0.0,"dataset_or_benchmark":1.0,"position_or_commentary":0.0}},"contribution_statement":{"p":1.0,"text":"To capture this setting, we introduce MemoryArena, a unified evaluation gym for benchmarking agent memory in multi-session Memory-Agent-Environment loops.","location":"abstract s4"}},"coverage":{"tier":"abstract","levels_run":["P abstract"],"not_examined":["full text","figures and tables","references and citing sentences","relations to other papers"],"sources_read":["arXiv listing abstract v2"],"levels_not_run":["S sections","T body sentences","C citations","R cross-paper relations"],"sentences_examined":8,"sentences_in_abstract":8},"measured_accuracy":{"url":"https://pith.science/accuracy","measured_on":"2026-10-02","method":"two independent model families labelled a fresh random sample blind; the figure is how often the published label matches their consensus","sentence_role_agreement":0.92,"main_claim_agreement":0.908,"typed_citation_agreement":0.933,"typed_citation_agreement_95_interval":[0.917,0.945],"review_comments_accurate":0.758,"review_comments_accurate_95_interval":[0.684,0.82],"review_caught_admitted_error_found_or_related":0.448,"review_comments_judged":{"judged_by":["gpt-5.6-luna","grok-4.7"],"judged_against":"Pith's own text copy of each paper: text extracted from the PDF, the first 90,000 characters","measured_on":"2026-10-04","accurate":113,"decided":149,"sampled_reviews":200,"split_by_lane":false,"by_lane":"https://pith.science/accuracy#by-lane"}}}