{"id":"b1919760-546b-4c31-91c0-c2d9e45e7fb8","arxiv_id":"2606.15504","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.5,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"A multi-agent clinical framework (CDA/TEA/CEMA) with memory/model/code self-evolution and a safety sandbox shows gains on synthetic error-propagation cases and mixed MedBench results after fine-tuning a 1.5B model.","lead":"VIBEMed is a three-agent clinical decision system that diagnoses, plans treatment, and tries to improve itself from clinician feedback and outcomes. It matters if hospitals want AI that learns from real cases without full retraining, but the evidence is mostly synthetic cases and a small model with mixed results.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.5","headline":"Synthetic LLM-generated cases + LLM judges + 1.5B proxy do not secure the claim of superior real-world clinical decision support.","rationale":"The reader correctly isolates the weakest assumption: synthetic Claude cases + LLM-as-judge + 1.5B proxy are treated as stand-ins for real clinical reliability and for transfer to larger models. That assumption is load-bearing for every superiority number cited in the abstract and §3. The multi-agent cascade result is a useful engineering signal under controlled error injection, and the architecture (role prompts, three-tier safety, hierarchical memory, sandbox isolation) is coherent; none of that is disputed. But the paper’s own strongest claim is empirical superiority for complex integrated/longitudinal decisions and real-world feasibility (oncology example is only a UI walkthrough). Those claims remain conditional on evidence that has not yet been produced. No stronger internal inconsistency or hidden mathematical flaw is present; the concern is purely evidential. Therefore the reader’s CONDITIONAL verdict and high correctness_risk stand; no upgrade or downgrade is warranted. A single concrete re-evaluation on real cases + larger backbone would settle whether the concern lands.","tokens_in":14484,"tokens_out":691,"duration_ms":6074,"concrete_test":"Re-run the exact §3.1 four-pipeline protocol on a held-out set of ≥50 real de-identified multi-morbidity/oncology cases with known clinician-adjudicated upstream errors and final outcomes; replace Qwen scoring with blinded clinician 1–5 ratings on the same eight metrics. Separately, apply the identical CEMA reflection + LoRA/DPO schedule to a ≥7B backbone and re-score MedTreat/MedPHM/MedDiag/MedAnalysis with the same LLM judges plus clinician spot-check. If cascade advantage falls below ~2 points or evolution gains reverse/vanish, the strongest claim does not hold.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (multi-agent cascade + CEMA evolution yields superior, safer CDS than static single models, especially for integrated/longitudinal decisions) rests almost entirely on two evaluations that do not measure that claim. §3.1 uses 30 Claude-Opus-4.6 synthetic cases with planted upstream errors, scored 1–5 by Qwen3.6 Plus on eight rubrics; Pipeline D (CDA+TEA) reaches 39.20/40 vs 35.13 for direct LLM. §3.2 fine-tunes a 1.5B DeepSeek-Distill-Qwen2.5 proxy with LoRA-SFT on ~4k treatment samples + 10k general data, then scores MedBench via three LLM judges; gains appear on MedTreat (+3.9) and MedPHM (+12.4) but regressions on MedAnalysis (−3.0) and MedDiag (−2.0). No real multi-institution records, no clinician-blinded outcomes, no prospective safety events, and no demonstration that the 1.5B gains (or the synthetic cascade scores) transfer to the production-scale models the architecture assumes. The paper itself flags these gaps in the limitations (§4). Without that transfer, the empirical superiority claim is unsupported even if the architecture is coherent.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes VIBEMed, a multi-agent clinical decision-support framework built around three specialized agents (CDA for diagnosis, TEA for treatment planning with guideline memory and three-tier safety constraints, and CEMA for distilling encounter outcomes into reflection data) plus a three-level self-evolution stack (memory, model via LoRA-SFT/DPO, and code) and an architecture-level safety sandbox (execution isolation and session-level memory isolation). The central claim is that this combination yields superior, safer, experience-driven clinical decisions relative to static single-model pipelines, especially for integrated and longitudinal tasks. Empirical support consists of (i) a 30-case synthetic multi-agent ablation (Table 1) in which the CDA+TEA cascade scores 39.20/40 versus 35.13 for a direct LLM, and (ii) evolution experiments on a 1.5B proxy model evaluated on MedBench tasks (Table 2), with gains on MedTreat, MedPHM, and MedDiffer but declines on MedAnalysis and MedDiag, plus a qualitative oncology interface demonstration.","tokens_in":14882,"tokens_out":940,"duration_ms":7762,"significance":"If the architecture and evolution claims hold under stronger evaluation, the work would be a useful systems contribution: it articulates a concrete multi-agent clinical workflow, a hierarchical memory design, staged SFT/DPO evolution with an LLM-as-judge promotion gate, and system-level isolation mechanisms that go beyond prompt-only medical agents. The multi-agent ablation is directionally informative about error propagation, and the explicit limitations section is appropriately candid. The manuscript does not, however, yet deliver machine-checked proofs, released code/data, or prospective multi-institution clinical outcomes; significance therefore remains conditional on validation that the synthetic and 1.5B-proxy results transfer to production-scale models and real clinical settings.","major_comments":[{"comment":"§3.1 and Table 1: The multi-agent superiority claim rests on 30 Claude-generated synthetic cases with planted upstream errors, scored 1–5 by another LLM (Qwen3.6 Plus) on eight rubrics. This design cannot establish real-world clinical reliability or safety; it mainly shows that a cascade can recover planted errors under LLM-judged rubrics. At minimum the paper should (a) report inter-judge agreement with clinicians on a subset, (b) include a non-LLM baseline or human-scored sample, and (c) temper abstract/conclusion language that implies real-world superiority from this table alone.","section":null},{"comment":"§3.2 and Table 2: Evolution is demonstrated only on a 1.5B DeepSeek-Distill-Qwen2.5 proxy with LoRA-SFT (~4k treatment + 10k general samples); gains on MedTreat (+3.9) and MedPHM (+12.4) are offset by regressions on MedAnalysis (−3.0) and MedDiag (−2.0). The claim that scaling laws make these gains predictive for larger production models is asserted but not tested. Either run the same protocol on at least one larger backbone, or reframe the result strictly as a small-model feasibility study and remove transfer language from the abstract and §3.2.","section":null},{"comment":"§2.3–2.4 and §4: The three-level evolution stack and architecture-level safety sandbox are load-bearing for the “safe continuous evolution” claim, yet neither is ablated or stress-tested (no rollback incidents, no isolation breach tests, no contribution of memory vs. model vs. code). The limitations section acknowledges this; the manuscript should either add hierarchical ablations / safety probes or substantially narrow claims that the sandbox “guarantees” reliable clinical operation.","section":null},{"comment":"Evaluation circularity (§3.1–3.2): Cases are LLM-generated, pipelines are LLM-based, promotion uses LLM-as-a-judge, and MedBench scoring uses three further LLMs. Reflection data for DPO is drawn from the same system’s interactions. Without an external clinician-labeled holdout or prospective outcome measure, the superiority claim for “experience-driven clinical decision support” remains weakly grounded. A concrete fix is a small clinician-blinded preference study on a fixed case set before and after evolution.","section":null}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful part of this paper is the packaging: CDA for structured diagnosis with uncertainty, TEA with guideline memory plus a three-tier safety filter, and CEMA that turns clinician adoption/rejection and outcomes into reflection data for memory, LoRA-SFT, and DPO. The architecture-level sandbox (dev/val/prod isolation, session-scoped memory, audit logs) is more carefully specified than most medical multi-agent write-ups. That design is real engineering work and worth reading if you care about hospital-deployable agent loops.\n\nWhat is new is mostly the integration and the “vibe medicine” framing, not the individual pieces. Multi-agent medical pipelines, RAG over guidelines, hierarchical memory, LoRA/DPO, and sandboxed code gen are all established. The paper’s contribution is a concrete end-to-end clinical workflow that tries to close the loop from encounter to persistent learning without constant human prompt engineering.\n\nThe soft spots are exactly where the stress-test points, and the authors mostly admit them. Table 1 (30 Claude-generated cases with planted upstream errors, scored by another LLM) is a clean engineering signal that the cascade can catch error propagation better than a single model, but it is not external validity. Table 2 on a 1.5B proxy shows real gains on MedTreat and MedPHM and real regressions on MedAnalysis and MedDiag; that undercuts any blanket “evolution improves clinical reasoning” claim. No released code/data, no real multi-site records, no clinician-blinded outcomes, and incomplete ablation of the three evolution layers and the safety mechanisms. Free parameters (adoption thresholds, dataset sizes, refresh cadences) are stated but not stress-tested.\n\nMath and citations are fine for a systems paper; nothing load-bearing is formally wrong. This is for people building clinical agent stacks or institutional LLM adaptation, not for someone looking for a settled clinical efficacy result. I would send it to peer review: the design is coherent enough and the gaps are fixable with better evaluation. Engage if you work on medical multi-agents; treat the empirical superiority claims as provisional until they run on real data and larger models.","headline":"Coherent multi-agent clinical stack with a clear evolution story, but the superiority claims rest on synthetic cases, LLM judges, and a 1.5B proxy with mixed results.","tokens_in":15585,"tokens_out":531,"would_cite":false,"duration_ms":4778,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"A three-agent medical AI that learns from its own cases across memory, model, and code can beat static systems on complex clinical decisions.","keywords":["Vibe Medicine","Large Language Model","Multi-Agent Systems","Self-Evolving Agents","AI Safety","Clinical Decision Support","Precision Medicine"],"falsifier":"A prospective multi-center trial on real patient encounters that measures whether the full evolved VIBEMed pipeline reduces clinically unsafe recommendations and improves clinician-adopted plan quality relative to a frozen single-model baseline of the same backbone, with independent human expert review of safety and accuracy.","tokens_in":15359,"feed_emoji":"🩺","tokens_out":638,"duration_ms":5807,"temperature":0.7,"pith_summary":"Most medical AI systems are frozen after training: they cannot turn real patient outcomes, clinician rejections, and past failures into lasting improvement. This paper introduces VIBEMed, a three-agent pipeline that mimics clinical workflow—one agent diagnoses, one plans treatment under safety constraints, and a third turns completed encounters into reusable experience. That experience drives continuous updates at three levels: hierarchical clinical memory for immediate adaptation, staged model fine-tuning for lasting capability gains, and automated code changes for new functions—all behind an architecture-level safety sandbox that isolates execution environments and patient sessions. On thirty synthetic complex cases the full cascade scores higher and more consistently than a single model or either agent alone, and after evolution a small backbone improves on several MedBench tasks that demand integrated treatment planning and long-term personalization. The authors present this as a concrete route from static medical AI toward systems that accumulate clinical experience the way clinicians do.","feed_headline":"Medical AI that learns from its own cases beats static systems","feed_subtitle":"Three agents plus memory-model-code evolution improve complex clinical decisions under a safety sandbox","key_machinery":"The three-level self-evolution mechanism (memory-level hierarchical storage and retrieval, model-level staged LoRA-SFT then DPO from CEMA reflection datasets, code-level sandbox-validated generation) together with the CDA–TEA–CEMA role pipeline and session-isolated safety sandbox.","core_discovery":"VIBEMed shows that a multi-agent clinical decision system with an explicit self-evolution loop—Clinical Diagnostic Agent, Therapeutic Execution Agent, and Clinical Evolution Manager Agent updating memory, model parameters via SFT/DPO, and code—outperforms static single-model pipelines on complex cases that require integrated reasoning and longitudinal planning, while an architecture-level safety sandbox keeps updates and patient data isolated.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Self-evolving multi-agent system beats static clinical AI","Three agents update memory and models for better decisions","VIBEMed learns from cases to outpace fixed medical pipelines","Clinical agents evolve via feedback for longitudinal planning","Sandboxed multi-agent loop improves complex treatment choices"],"cache_read_input_tokens":128,"weakest_assumption_plain":"That synthetic cases with planted errors scored by another LLM, plus evolution experiments on a 1.5B proxy model judged the same way, are faithful enough proxies for real multi-institution clinical reliability and for gains that will transfer to larger production models.","fun_headline_variants_meta":{"raw":{"variants":["Self-evolving multi-agent system beats static clinical AI","Three agents update memory and models for better decisions","VIBEMed learns from cases to outpace fixed medical pipelines","Clinical agents evolve via feedback for longitudinal planning","Sandboxed multi-agent loop improves complex treatment choices"]},"model":"grok-4.5","effort":"low","cost_usd":0.003468,"raw_usage":{"total_tokens":1190,"prompt_tokens":822,"num_sources_used":0,"completion_tokens":79,"cost_in_usd_ticks":34680000,"prompt_tokens_details":{"text_tokens":822,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":289,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":822,"tokens_out":79,"duration_ms":3622,"temperature":1.0,"reasoning_tokens":289,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T15:10:23.127470+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"A prospective multi-center trial on real patient encounters that measures whether the full evolved VIBEMed pipeline reduces clinically unsafe recommendations and improves clinician-adopted plan quality relative to a frozen single-model baseline of the same backbone, with independent human expert review of safety and accuracy.","supporting_citations":[],"review_version":1}