{"id":"fa54ae64-e50c-43c9-9165-7fa774c6b7bb","arxiv_id":"2607.09322","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A MIMIC-IV-based long-horizon medical-agent benchmark finds LLMs use explicit timestamps well, fail at implicit temporal reasoning, and decide mainly from immediate context despite memory tools.","lead":"LongMedBench turns MIMIC-IV records into multi-visit event streams and tests medical LLM agents on fact recall, timeline reconstruction, and next-step clinical decisions. It shows models handle explicit timestamps well but struggle with implicit time and still decide mainly from the current context.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Decision-suite proxy validity is the load-bearing soft spot for the independence-of-history claim.","rationale":"The reader correctly isolates the weakest link: the synthetic decision tasks are the only bridge from the observed flat ablation curves to the claim that decision quality depends on immediate context rather than retrieved history. That assumption is load-bearing for the paper’s most distinctive negative result; the explicit-timestamp and retrieval findings are better supported by Tables 2–3 and do not require the same proxy leap. Because the concern is about external validity of the decision suite rather than an internal contradiction or data-processing error, the appropriate stance remains CONDITIONAL (release artifacts + tighten clinical validation of T3), not REJECT. No stronger technical flaw (e.g., leakage of future events past timestamp T, or non-reproducible filtering) is evident from the text that would overturn the claim more directly. The concrete clinician re-rating test would settle whether the proxy is adequate without requiring a full re-benchmark.","tokens_in":11548,"tokens_out":587,"duration_ms":6720,"concrete_test":"On a stratified 50-patient subset, have two clinicians independently rate (a) clinical appropriateness of the T3 multiple-choice options and (b) whether the gold next action is uniquely determined by long-horizon history vs. the immediate visit alone. Re-score model outputs only on the high-agreement, history-dependent items; if the Table-5 flatness (no gain from n=0\to5 or RAG/Mem0) disappears or reverses, the independence claim is an artifact of the synthetic suite.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper’s central experimental claim (Tables 4–5, §3.2, Conclusion) is that long-horizon decision accuracy is largely independent of historical memory volume and is instead governed by the model’s immediate context. That claim rests on T3-N / T3-A / T3-D being faithful proxies for clinical next-step quality. Those tasks are constructed by (i) LLM-rewriting the current visit’s event stream into action–feedback dialog pairs for contextual memory M_C, (ii) sampling a ground-truth event and offering multiple-choice options, and (iii) applying a 24 h time-decay credit rule for near-future actions (§2.2, Fig. 2). If the options are too easy to guess from local cues, if the LLM rewrite injects its own clinical priors, or if the 24 h window systematically rewards recency rather than true longitudinal integration, then the flat performance curves under visit injection / RAG / Mem0 become an artifact of task construction rather than evidence that historical memory is unused. The factual-QA and temporal-reasoning suites do not rescue this, because they measure different capabilities.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper introduces LongMedBench, a MIMIC-IV-based benchmark for evaluating LLM medical agents on long-horizon clinical reasoning. A three-stage pipeline filters patients with ≥15 hospitalizations (reported as 335/355), builds visit-level event streams (avg. 19.72 visits, 44.91 events/visit), and constructs three memory modules (Note, Event, Contextual). An evaluation taxonomy covers (1) factual QA under explicit timestamps vs. relative positioning, (2) temporal reasoning via visit cloze, visit sorting, and joint sorting scored by Kendall’s τ, and (3) long-horizon decision-making (next-action, argument, and discharge prediction) with a 24 h time-decay rule. Experiments with gpt-5-mini, deepseek-v3.2, and qwen-turbo under naive long context, RAG, and Mem0 show strong use of explicit timestamps, sharp drops on implicit visit-level ordering, retrieval gains from memory systems, and decision accuracy that remains largely flat under added history or retrieval and is dominated by immediate context.","tokens_in":11935,"tokens_out":1374,"duration_ms":11681,"significance":"If the reported patterns hold, LongMedBench fills a clear gap relative to short-context medical agent and knowledge-QA benchmarks (Table 1) by providing real EHR longitudinal trajectories and a progressive taxonomy that separates fact retrieval, temporal reconstruction, and next-step planning. The reproducible MIMIC-IV pipeline, multi-granularity memory design, and concrete findings on explicit vs. implicit time and the limited transfer of RAG/Mem0 gains to decision tasks are useful contributions for medical-agent evaluation. Strengths include grounding in real admissions/notes, external metrics (Kendall’s τ, accuracy), and systematic ablations over window size, memory type, and architecture (Tables 2–5).","major_comments":[{"comment":"§2.2 and Fig. 2 (Long-Horizon Decision Making): The central claim that decision quality is largely independent of historical memory volume and depends on immediate context (Tables 4–5, §3.2, Conclusion) rests on T3-N/T3-A/T3-D as proxies. These tasks use LLM-rewritten action–feedback pairs for M_C, multiple-choice options sampled from the event stream, and a 24 h time-decay credit rule. The manuscript does not validate that options cannot be solved from local cues alone, that the rewrite does not inject model priors, or that the 24 h window does not systematically reward recency over longitudinal integration. Without such checks (e.g., no-history baselines beyond the n=0 row, human clinician agreement, or ablation of the decay window), the flat curves under visit injection/RAG/Mem0 risk being task-construction artifacts rather than evidence that history is unused.","section":null},{"comment":"Abstract vs. §2.1: Patient count is stated as 335 in the abstract and 355 after filtering in the pipeline text (and Fig. 1). This inconsistency affects the claimed scale of the benchmark and should be resolved with a single definitive N and a clear inclusion flowchart.","section":null},{"comment":"§2.1 (Contextual Memory) and §3.1: Contextual memory M_C is produced by rewriting the current visit’s events into assistant/user dialog pairs via an LLM. The paper does not specify which model is used for rewriting, the prompt, or leakage controls beyond the timestamp cutoff T. Because the same family of models is later evaluated as agents, rewrite-induced style or clinical priors could inflate or distort decision scores; this needs explicit documentation and, ideally, a sensitivity check with a different rewriter.","section":null}],"minor_comments":[{"comment":"Table 1 caption and body: Abbreviations Long./Ctx./EHR/Dec./Temp. are defined in footnotes; ensure consistent capitalization and that “Long.” clearly means multi-visit longitudinal reasoning rather than long context alone.","section":null},{"comment":"Fig. 1 and §2.1: Typographical noise in the figure text (e.g., “pr ocedur”, “pharmac”, “Not e”) and the 355 vs. 335 discrepancy should be cleaned for camera-ready quality.","section":null},{"comment":"Table 2: Lab-T / Lab-F / Lab-O notation is only partially expanded in footnotes; a short legend in the table caption would improve readability.","section":null},{"comment":"§3.2 Temporal Reasoning: “regradless” → “regardless”; also clarify whether Kendall’s τ is computed only on correctly paired items or on the full permutation in joint sorting.","section":null},{"comment":"References and model access dates (e.g., GPT-5 mini, text-embedding-3-small “accessed 22 Feb 2026”) should be double-checked for consistency with the arXiv submission date and public availability.","section":null},{"comment":"§2.2 Factual QA: The window construction M_{(i-m,i+m+1)_E} with Vi placed in the middle is a good design against recency bias; stating the exact token lengths or event counts for m∈{1,2,3,∞} would help reproducibility.","section":null}],"recommendation":"major_revision","confidential_remarks":"The decision-suite proxy issue is the main load-bearing soft spot; if the authors can add even modest validation (clinician spot-check of a sample of T3 items, or a pure local-context control that isolates history), the paper becomes much stronger. Scope is appropriate for a methods/benchmark venue in medical AI; novelty relative to MedAgentBench/AgentClinic/DiagBench is real but incremental. No integrity red flags beyond the 335/355 inconsistency and the opaque LLM rewrite step."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is a real benchmark contribution for medical agents, not another short-session tool-use toy. The useful finding is the split between retrieval and decisions—explicit timestamps work, implicit visit-level time is hard, and RAG/Mem0 help fact tasks while next-step scores stay flat when you inject more history.\n\nWhat is new is the combination, not any single piece. They turn multi-visit MIMIC-IV into event streams (roughly 20 visits/patient), define three memory granularities (note/event/contextual), and run a progressive taxonomy: explicit vs relative fact QA, visit cloze/sorting/joint sorting with Kendall’s τ, then next-action/argument/discharge. Table 1 positioning against MedAgentBench, AgentClinic, DiagBench, etc. is fair. The pipeline is coherent, metrics are external, and Tables 2–5 support the qualitative pattern they claim. That negative result on “more history ≠ better decisions” is worth having in the literature.\n\nSoft spots, in proportion. Patient count flips between 335 and 355—sloppy but not load-bearing. Model coverage is thin (qwen-turbo, deepseek-v3.2, gpt-5-mini). The real soft spot is the decision suite: contextual memory is LLM-rewritten from the same stream, options are multiple-choice, and T3-N/A use a 24h time-decay credit. If local cues or rewrite priors make history unnecessary, the independence-of-history claim is partly an artifact. Factual QA and temporal suites do not fully rescue that claim; they measure different skills. No clear public release of processed splits/code in the text also limits immediate reuse.\n\nWho it is for: people building or evaluating long-context medical agents and memory systems. Not a methods breakthrough; a yardstick with an honest negative result. I would send it to peer review. Tighten the decision proxy (or validate it clinically), fix the count inconsistency, release artifacts, and it is a solid workshop/conference piece. Worth engaging if you work in this lane; skim the decision tables with the proxy caveat in mind.","headline":"Useful MIMIC-IV long-horizon agent benchmark with a real negative result on memory tools; decision-suite proxy is the soft spot, not the pipeline.","tokens_in":12541,"tokens_out":539,"would_cite":true,"duration_ms":10857,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Medical AI agents retrieve facts from long records but still decide mainly from what is in front of them.","keywords":["medical agents","long-horizon reasoning","EHR","temporal reasoning","clinical decision-making","long-context LLMs","MIMIC-IV","agent memory"],"falsifier":"Run the same models on a held-out set of real clinician next-step annotations for the same patient trajectories; if decision accuracy then rises sharply with more retrieved history (or falls when history is ablated), the claim that decisions depend mainly on immediate context would be falsified.","tokens_in":12425,"feed_emoji":"🏥","tokens_out":878,"duration_ms":9452,"temperature":0.7,"pith_summary":"Real clinical care is longitudinal: doctors integrate years of visits, labs, imaging, and treatment responses. Existing tests of medical language-model agents mostly check short-context knowledge or tool calls and miss that demand. This paper builds LongMedBench from real hospital records, turning hundreds of multi-visit patient trajectories into event streams and graded memory layers so agents can be tested over long horizons. It scores three abilities in order of difficulty: retrieving facts with or without timestamps, reconstructing temporal order when time cues are implicit, and choosing next clinical actions under long history. The experiments show that current models handle explicit timestamps and retrieval well, especially with memory tools, yet struggle with implicit visit-level ordering; decision accuracy stays largely driven by the immediate working context rather than how much history is retrieved.","feed_headline":"Medical AI still decides from what is in front of it","feed_subtitle":"LongMedBench shows models retrieve long records yet barely use them for next-step clinical planning","key_machinery":"LongMedBench: a MIMIC-IV-derived pipeline that converts multi-visit EHRs into time-series event streams and three graded memory modules (note summaries, flattened event streams, and current-visit contextual dialogs), paired with a progressive evaluation taxonomy of factual QA, temporal sorting/cloze tasks, and next-action/argument/discharge decision tasks with time-decay scoring.","core_discovery":"Under LongMedBench, recent long-context language models can exploit explicit timestamps for event-level ordering and fact retrieval, but performance collapses on implicit visit-level temporal reasoning; retrieval-augmented and agent-memory systems improve information-retrieval tasks, while long-horizon decision-making accuracy remains highly dependent on the model's immediate context rather than the volume of retrieved historical information.","pith_inferences":["The same pattern—strong explicit retrieval, weak implicit temporal integration—likely appears in other longitudinal domains such as chronic-disease management or multi-episode legal case files.","If decision quality is truly context-local, training or scaffolding that forces models to justify actions against distant visits may be more useful than simply enlarging memory stores.","A natural next experiment is to replace the synthetic options with open-ended free-text next-step plans scored by clinicians, testing whether the immediate-context dependence survives a less constrained format."],"forward_implications":["Benchmarks that only test short-context QA or tool use will overstate readiness of medical agents for real multi-visit care.","Improving retrieval or external memory alone will not close the gap on decision tasks if the model cannot integrate history into planning.","Explicit timestamp cues mask a deeper weakness: models still fail at ordering visits from clinical progression alone.","Future agent designs must target cross-session integration, not only longer context windows or better search.","The three-level memory design can be reused to measure whether new architectures actually use long-term clinical trajectories."],"fun_headline_variants":["LongMedBench: LLMs use timestamps but fail implicit clinical timelines","RAG aids retrieval; decisions stay stuck on immediate context","Medical agents ignore most history when planning next steps","Explicit time works; visit-level temporal reasoning collapses","Long records retrieved yet barely shape multi-visit decisions"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The synthetic multiple-choice next-action, argument, and discharge questions—built from LLM-rewritten event trajectories and a 24-hour time-decay score—are valid stand-ins for real long-horizon clinical decision quality.","fun_headline_variants_meta":{"raw":{"variants":["LongMedBench: LLMs use timestamps but fail implicit clinical timelines","RAG aids retrieval; decisions stay stuck on immediate context","Medical agents ignore most history when planning next steps","Explicit time works; visit-level temporal reasoning collapses","Long records retrieved yet barely shape multi-visit decisions"]},"model":"grok-4.5","effort":"low","cost_usd":0.005664,"raw_usage":{"total_tokens":1534,"prompt_tokens":789,"num_sources_used":0,"completion_tokens":65,"cost_in_usd_ticks":56640000,"prompt_tokens_details":{"text_tokens":789,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":680,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":789,"tokens_out":65,"duration_ms":6576,"temperature":1.0,"reasoning_tokens":680,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T03:52:57.770926+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run the same models on a held-out set of real clinician next-step annotations for the same patient trajectories; if decision accuracy then rises sharply with more retrieved history (or falls when history is ablated), the claim that decisions depend mainly on immediate context would be falsified.","supporting_citations":[],"review_version":1}