{"id":"86f64a11-81e9-47a5-b636-3c90c051fdc3","arxiv_id":"2607.22566","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A new EHR-grounded benchmark of 100 long multi-session patient dialogues shows LLMs are consistently worse at cross-admission reasoning than at single-admission QA.","lead":"This paper introduces a new medical benchmark built from real deidentified hospital records, with 100 synthetic multi-visit patient conversations and 17,892 questions. It finds that language models answer questions about a single hospital stay fairly well but consistently struggle when the answer requires connecting evidence across several admissions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-admission gap is confounded with question type: MedLoCoMo's scope contrast does not isolate evidence distribution as the cause of difficulty.","rationale":"The reader's weakest assumption concerned QA label correctness; my concern is a construct-validity confound in the scope contrast. Both threaten the interpretation of the central claim, but the question-type confound is visible directly from Table 5 and Section 3.5: the single-admission and cross-admission item sets differ in question type, not only in evidence span. This makes the headline 'cross-admission reasoning is harder than localized evidence use' an overstatement as a diagnostic claim. The flattening test would settle whether the gap is driven by evidence distribution or by the reasoning operations encoded in the cross-admission question types. I keep the verdict CONDITIONAL because the benchmark is still useful and the issue is addressable with additional controlled comparisons, but the central claim should be reworded or the benchmark should add matched question types.","tokens_in":19985,"tokens_out":10013,"duration_ms":111474,"concrete_test":"Flattening test on the released data: take a stratified sample (e.g., 300 items) of cross-admission answerable QA (LP/CAC/FP). For each, concatenate all cited evidence turns from the multiple admissions into one synthetic single-admission session, keeping the question, answer, and total token count unchanged; run the same baseline set (at least GPT-5.1, Qwen3.5-27B, Gemma3-27B) on both original multi-admission and flattened single-admission contexts. If the J/F1 gap largely persists, question type is the driver and the diagnostic claim needs qualification; if the gap collapses, evidence scope is confirmed. As a complementary check, generate single-admission LP/CAC/FP questions from one admission's turns and compare within type.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (Abstract; §4) is that cross-admission reasoning is consistently harder than localized evidence use, even with long contexts, memory, or retrieval. But the two evidence scopes are not matched in question type. §3.5 defines single-admission answerable types as medical_reasoning and care_plan_rationale, while cross-admission answerable types are longitudinal_progression, cross-admission_comparison, and frequency_pattern. Table 5 shows these types are not comparable: e.g., GPT-5.1 judge accuracy is 87.2/89.7 on MR/CPR but 45.1/35.7/34.1 on LP/CAC/FP. The reported single-to-cross gap (Table 2; Figure 4) may therefore reflect the extra aggregation, comparison, or counting operations required by cross-admission question types, not the fact that evidence spans multiple admissions. Retrieval and memory methods that improve evidence selection would not be expected to close a question-type gap, so the conclusion that context length alone does not remove the gap does not establish that distributed-evidence integration is the bottleneck. As released, the benchmark does not cleanly separate local evidence use from cross-admission integration.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"MedLoCoMo is a benchmark for evaluating LLMs on longitudinal, patient-specific clinical reasoning. It converts MIMIC-IV and MIMIC-IV-Note records into 100 patient timelines with synthetic doctor–patient conversations per admission, then generates short-answer QA items in three scopes: single-admission answerable, cross-admission answerable, and adversarial unanswerable. Evaluations on general-purpose, medical, and memory/retrieval baselines show that cross-admission judge accuracy is consistently lower than single-admission, which the paper interprets as evidence that context length, memory, and retrieval do not remove the gap.","tokens_in":20346,"tokens_out":5168,"duration_ms":52351,"significance":"The benchmark addresses an important missing evaluation dimension and the release is unusually transparent: code, patient-level artifacts, evidence links, and generation prompts are provided. The pipeline is staged and includes automated schema/evidence checks plus a sampled human audit. The paper also carefully distinguishes metrics and warns against overreading Score. If QA labels are valid, MedLoCoMo would be a valuable diagnostic for separating local evidence use, cross-admission integration, and abstention. The main empirical pattern is qualitatively consistent across models. However, the headline claim is currently confounded with question type and depends on lightly validated automatically generated labels; these issues must be addressed before the benchmark can support the study's conclusions.","major_comments":[{"comment":"The central claim that cross-admission reasoning is harder than localized evidence use is confounded with question type. Single-admission answerable items are only medical_reasoning and care_plan_rationale; cross-admission answerable items are only longitudinal_progression, cross-admission_comparison, and frequency_pattern. Table 5 shows large type-level differences (e.g., GPT-5.1 J: 87.2/89.7 vs 45.1/35.7/34.1), so the cross-vs-single gap in Table 2 and Figure 4 may reflect the extra aggregation/comparison/counting operations rather than evidence distribution. The conclusion that retrieval and memory cannot close the gap does not follow unless question type is controlled. Please generate matched question types across scopes (e.g., single-admission frequency_pattern and cross-admission medical_reasoning) or explicitly adjust for question type in the analysis.","section":"§3.5 and Table 5"},{"comment":"Label correctness is load-bearing and under-validated. Only about 300 QA items (~1.7%) were human-audited; automated checks (A.8) verify schema, evidence citations, monotonic timestamps, and adversarial gold-answer formatting, but they do not verify that answerable answers are semantically entailed by the cited evidence or that adversarial questions are truly unanswerable. Because the QA items are LLM-generated, systematic label errors would make the reported cross-admission gap an artifact of the generator rather than a property of model reasoning. Please provide a larger stratified audit with confidence intervals, inter-annotator agreement, public release of audit materials, and/or automated entailment and unsupportedness checks.","section":"§3.5, §A.8"},{"comment":"No uncertainty quantification is reported. All results are point estimates; there are no standard errors, confidence intervals, or significance tests. With 100 patients and many correlated QA items per patient, differences such as GPT-5.1 vs Qwen3.5-27B J (64.2 vs 56.1) and several memory-method comparisons in Table 3 may be within noise. Report patient-level bootstrap confidence intervals or a mixed-effects analysis, at least for the headline cross-admission gap and the memory/retrieval comparisons.","section":"§4.1, Tables 2–3"}],"minor_comments":[{"comment":"The pipeline figure is difficult to read; text is small and some box labels appear garbled. Please provide a high-resolution vector version with readable fonts.","section":"Figure 2"},{"comment":"The claim that preliminary judge choices produced 'completely consistent binary decisions together with human review' needs quantitative support: number of cases, percent agreement, or Cohen's kappa.","section":"Appendix B.2"},{"comment":"The Score formula is typeset in a way that obscures the denominator. Please display the formula cleanly with explicit numerator and denominator.","section":"Appendix B.1"},{"comment":"The 'Time interval' for MedLoCoMo is listed as 'few years'. Report actual statistics (median and range) in the table or main text, since temporal span is a key claimed property.","section":"Table 1"},{"comment":"The phrase 'grounded doctor–patient conversations' is strong for LLM-generated dialogues whose grounding is verified only on a small human sample. Consider wording such as 'evidence-derived' or explicitly qualifying the grounding claim.","section":"§3.3"},{"comment":"Recommended turn range, packet truncation thresholds, and related generation parameters are free choices. Please report or release the exact values used so the construction is fully reproducible.","section":"Appendix A.1"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and release are potentially valuable for the clinical NLP / long-context evaluation community. The main risk is the question-type confound, which is not merely a presentation issue: it requires either new matched QA construction or a substantially re-framed analysis. The validation and statistical concerns are also substantial but addressable. I recommend major revision rather than rejection, as the underlying resource seems useful and the pipeline is transparent."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First things first: MedLoCoMo is a real contribution. It fills an actual gap—patient-level, multi-admission medical dialogue QA with evidence-linked items—and it ships a nontrivial artifact: 100 timelines, ~18k QA items, code, and a transparent construction pipeline. The sampled human audit and the automated schema/evidence checks are more than most benchmark papers do. The baseline sweep is broad, and the memory/retrieval case study is a sensible addition.\n\nThe soft spot is the one the stress-test note flags, and it is real. The paper's headline claim—cross-admission reasoning is harder than localized evidence use—is built on a comparison that confounds evidence scope with question type. Single-admission answerable items are medical_reasoning and care_plan_rationale; cross-admission answerable items are longitudinal_progression, cross-admission_comparison, and frequency_pattern. Those latter types require aggregation, comparison, or counting even if the evidence were in one place. Table 5 shows the gap is already visible in GPT-5.1's judge accuracy: 87/89 on the local types, 45/35/34 on the cross types. So the reported single-to-cross difference is at least partly a question-type difficulty gap, not purely a distributed-evidence integration gap. That weakens the abstract's causal reading. It does not, however, invalidate the benchmark as a resource: the QA types themselves are worth testing, and the evidence references make it reproducible.\n\nOther concerns are minor. The human audit covers a small sample, and no confidence intervals or significance tests accompany the model comparisons. The LLM-judge reliability is asserted rather than quantified. Those are fixable in revision.\n\nIf I were the editor, I would send this to peer review. The benchmark fills a niche and the authors are transparent about limitations. But I would ask the authors to confront the confound—either by adding matched question types across scopes or by softening the claim to 'cross-admission question types are harder'—and to add basic statistics and a larger human audit. As is, the paper deserves a serious referee; the central diagnostic conclusion needs revision.","headline":"Useful new benchmark for longitudinal medical dialogue QA, but the headline claim about cross-admission difficulty is confounded by question type.","tokens_in":20752,"tokens_out":2504,"would_cite":true,"duration_ms":25585,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces MedLoCoMo, a benchmark of 100 multi-admission patient timelines, and reports that every evaluated model answers single-admission questions markedly better than cross-admission questions, regardless of context length or","keywords":["medical dialogue benchmark","long-context LLM evaluation","cross-admission reasoning","clinical QA","EHR","MIMIC-IV","abstention","synthetic data"],"falsifier":"Audit a large random sample of the 17,892 QA items, beyond the ~300 already reviewed, and count how often an adversarial question is actually answerable from the record or a cross-admission question is answerable from a single admission. If either error rate is substantial, or if re-evaluating on a human-verified subset removes the cross-admission gap, the central claim would be an artifact of the generator rather than a fact about model reasoning.","tokens_in":19877,"feed_emoji":"🩺","tokens_out":3762,"duration_ms":39234,"temperature":0.7,"pith_summary":"This paper introduces MedLoCoMo, a benchmark that tests whether large language models can reason over a patient's entire longitudinal record rather than one document or one visit. The benchmark is built from deidentified EHR records by turning each hospital admission into a synthetic doctor–patient conversation, chaining admissions into timelines, and attaching evidence-linked questions that are answerable from one admission, require several admissions, or are deliberately unanswerable. Across every evaluated baseline, models score far higher on single-admission questions than on cross-admission questions, and longer context windows, medical specialization, retrieval, and external memory all leave that gap intact. The authors argue this makes MedLoCoMo a diagnostic tool that separates local evidence use, cross-admission integration, and abstention — three abilities a useful long-context clinical system needs and that aggregate scores conflate.","feed_headline":"LLMs answer single-admission facts but miss cross-admission links","feed_subtitle":"A new 100-patient medical dialogue benchmark finds long context windows and memory tools don't close the reasoning gap.","key_machinery":"The load-bearing object is the patient-level timeline with evidence-linked QA items. Each patient is one benchmark instance; each hospital admission is one session; admissions are converted into grounded synthetic dialogues and then into short-answer questions whose evidence field cites specific admissions and turns. The QA split separates single-admission questions, cross-admission questions (requiring evidence from at least two admissions), and adversarial unanswerable questions that carry a canonical abstention answer. Evaluation reports token-level F1, LLM-judge accuracy for answerable items, abstention accuracy for adversarial items, and a weighted Score, with the components kept separa","core_discovery":"The central finding is that cross-admission reasoning is consistently harder than localized evidence use across the entire evaluated model suite. For example, GPT-5.1's judge accuracy falls from 88.4% on single-admission questions to 39.9% on cross-admission ones, and Qwen3.5-27B, the strongest single-admission extractor, shows the largest drop. Memory-augmented and retrieval baselines improve abstention and composite scores, but no method eliminates the cross-admission gap, indicating that the difficulty is not simply context-window length. MedLoCoMo's design — admission-level packets, chronological patient timelines, and evidence-linked QA with single-admission, cross-admission, and advers","pith_inferences":["If the same gap appears in human-verified or real clinical dialogue, it would argue for evaluating EHR assistants patient-wise rather than note-wise; per-note benchmarks would miss the hardest failures.","A testable extension is to stratify cross-admission questions by the temporal distance or number of intervening admissions between evidence turns; the paper's design would predict increasing difficulty as that distance grows.","Because conversations and questions are synthetic, the measured gap could be inflated by generation artifacts — e.g., if generated cross-admission questions are worded to be harder than single-admission ones. A random human-labeled subset would settle whether the gap is a property of model reasoning or of the generator.","The benchmark could double as a training signal: generating more cross-admission examples and directly optimizing linking could close the gap, but the study's abstention results warn that score gains from abstention can hide a remaining extraction deficit."],"forward_implications":["If the reported gap holds, context length alone is not a proxy for longitudinal clinical reasoning; models need mechanisms for selecting and linking evidence across encounters.","Medical specialization is not sufficient: several medical models abstain well but still extract cross-admission answers poorly, so domain tuning should be evaluated on span-connecting questions, not just knowledge recall.","Memory and retrieval systems should be judged by cross-admission answer quality, not by composite Score, since their main measurable benefit in this study is better abstention.","The benchmark's three-way diagnostic structure lets a developer attribute a system's failure to losing local facts, failing to connect admissions, or over-answering unsupported questions.","For a system to be useful in longitudinal care, it must do all three: extract local facts, connect them across admissions, and abstain — no single behavior substitutes for the others."],"fun_headline_variants":["Cross-admission reasoning trips even long-context LLMs","Memory tools fail to close LLM cross-admission gap","100-patient benchmark: LLMs miss links across admissions","Single-admission facts easy, cross-admission links hard for LLMs","New benchmark shows LLMs struggle with multi-session medical history"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The result stands on the assumption that the automatically generated QA labels are correct — that answerable questions are truly answerable from their cited evidence and adversarial questions are truly unanswerable — since only a sample of about 50 conversations and 300 questions was human-checked.","fun_headline_variants_meta":{"raw":{"variants":["Cross-admission reasoning trips even long-context LLMs","Memory tools fail to close LLM cross-admission gap","100-patient benchmark: LLMs miss links across admissions","Single-admission facts easy, cross-admission links hard for LLMs","New benchmark shows LLMs struggle with multi-session medical history"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000145,"raw_usage":{"total_tokens":1005,"prompt_tokens":723,"completion_tokens":282,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":212}},"tokens_in":467,"tokens_out":282,"duration_ms":3458,"temperature":1.0,"reasoning_tokens":212,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T12:37:59.260939+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Audit a large random sample of the 17,892 QA items, beyond the ~300 already reviewed, and count how often an adversarial question is actually answerable from the record or a cross-admission question is answerable from a single admission. If either error rate is substantial, or if re-evaluating on a human-verified subset removes the cross-admission gap, the central claim would be an artifact of the generator rather than a fact about model reasoning.","supporting_citations":[],"review_version":1}