{"id":"d7363e13-bc2e-4838-ac6d-99d5bb7934e0","arxiv_id":"2603.14771","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"OpenHospital is an interactive physician-patient multi-agent arena that improves clinical metrics via ground-truth reflection and reports cooperative behaviors as evidence of evolving LLM collective intelligence.","lead":"OpenHospital is a multi-agent hospital simulation where AI physician agents improve diagnosis and treatment by interacting with synthetic patient agents over many cases. It supplies metrics and a training loop meant to grow and measure collective intelligence in LLM agents beyond static datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Ground-truth-guided reflection after every case makes the reported evolution look more like supervised individual fine-tuning than spontaneous data-in-agent-self CI.","rationale":"The Reader correctly isolates the weakest link: evolution is scaffolded by ground-truth oracles and synthetic patients validated only by LLM judges. My stress-test confirms that this is load-bearing rather than peripheral. Without an oracle-free control, the quantitative trajectories in Figures 4–5 and the qualitative consultation example in Figure 6 cannot distinguish genuine collective intelligence from ordinary supervised individual improvement. The paper already acknowledges related limitations (unimodal, simplified temporal dynamics) but does not address this core causal ambiguity. The recommended concrete ablation is therefore decisive: its outcome either salvages or undercuts the strongest claim. Because the engineering contribution remains solid and the missing control is fixable, the Reader’s CONDITIONAL verdict is unchanged; only clearer language and the ablation are required before the CI framing can be accepted.","tokens_in":10825,"tokens_out":528,"duration_ms":5923,"concrete_test":"Re-run the identical 22-batch schedule with the reflection oracle ablated (or replaced by a noisy/self-generated critique that never sees ground truth). If Examination Precision, Diagnostic Accuracy and Treatment Plan Alignment gains shrink by more than half and cross-departmental consultation frequency collapses, the CI claim is unsupported; if the gains remain comparable, the concern is refuted.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (significant metric gains + spontaneous cooperation = genuine CI from data-in-agent-self) rests on the closed-loop reflection mechanism of Section 4.2: after each case agents perform multi-dimensional self-critique that synthesizes diagnostic accuracy against ground truth, examination efficiency, and therapeutic safety. Because the oracle is available at every step, the observed rises (Examination Precision 45.05%→61.31%, Diagnostic Accuracy 48.11%→57.34%, Treatment Plan Alignment 58.49%→61.52%) and the token decline can be explained by ordinary supervised improvement of individual agents rather than by emergent collective intelligence arising from physician–patient interaction alone. The paper supplies no ablation that removes or degrades this ground-truth signal, so the “data-in-agent-self” and “CI” interpretations remain untested.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper introduces OpenHospital, an interactive multi-agent arena for evolving and benchmarking LLM-based collective intelligence (CI) in a clinical setting. Physician agents interact with synthetic patient agents under a “data-in-agent-self” paradigm, using a multi-stage pipeline (DeepSeek-v3.1) to generate 12,000 patient records with comorbidities and long-tail diseases across 19 departments. A baseline of 38 physician agents (Agent-Kernel, Qwen3-Next-80B) is trained over 22 batches of ~500 cases each; after every case agents perform multi-dimensional self-critique against ground-truth diagnoses, examinations and treatment guidelines. Reported gains are Examination Precision 45.05%→61.31%, Diagnostic Accuracy 48.11%→57.34%, Treatment Plan Alignment 58.49%→61.52%, with declining total input tokens; qualitative case studies illustrate refined individual reasoning and spontaneous cross-departmental consultation. The authors claim the arena both fosters genuine CI and supplies rigorous dual metrics of medical proficiency and system efficiency.","tokens_in":11062,"tokens_out":722,"duration_ms":6077,"significance":"If the central claim holds—that physician–patient interaction plus closed-loop reflection produces genuine, quantifiable collective intelligence rather than ordinary supervised improvement—the work would supply a reusable, privacy-safe evolutionary arena and a multi-dimensional clinical benchmark that current static MAS evaluations lack. Strengths include a large synthetic comorbidity-rich dataset, explicit multi-metric tracking (examination, diagnosis, treatment, tokens), open-source Agent-Kernel linkage, and qualitative evidence of peer consultation. These contributions would be useful to the multi-agent and medical-AI communities even if the strongest CI interpretation requires further controls.","major_comments":[{"comment":"§4.2 Baseline / closed-loop reflection: after every case agents receive multi-dimensional self-critique that explicitly synthesizes diagnostic accuracy against ground truth, examination efficiency and therapeutic safety. The same ground truth defines the three Medical Capability metrics in §4.1. Without an ablation that removes or degrades this oracle signal (or a pure interaction-only control), the reported trajectories (Fig. 4) and token decline (Fig. 5) are equally consistent with ordinary supervised individual improvement; the “data-in-agent-self CI” interpretation therefore remains untested and load-bearing for the central claim.","section":null},{"comment":"§4.3 and Fig. 6 (cooperative behaviors): the action space already includes multi-agent consultation (§4.2). The single qualitative example of Infectious-Diseases → Cardiology consultation does not establish that cooperation is spontaneous or necessary rather than an enabled primitive. A non-consulting or single-agent control is required to support the claim that OpenHospital’s collaborative necessity drives emergent CI.","section":null},{"comment":"§3 and §4.1 evaluation: both patient-agent validation (Medical Consistency 4.4113, Accuracy/Relevance/Persona scores) and Treatment Plan Alignment rely on LLM-as-judge (GPT-5.2 / Baichuan-M2). No human clinician inter-rater reliability, no error bars or statistical tests on the 22-batch trajectories, and no comparison against a non-LLM clinical gold standard are reported. This weakens the claim that the metrics constitute a “rigorous” benchmark of medical proficiency.","section":null}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing to know is that OpenHospital is a competent, reusable clinical multi-agent simulator that actually tracks agents over 22 batches and reports rising exam precision, diagnostic accuracy, and treatment alignment while tokens fall. The second is that the load-bearing “collective intelligence from data-in-agent-self” claim is weaker than the abstract suggests: after every case the physicians get multi-dimensional self-critique against ground-truth diagnoses, exams, and guidelines. That is ordinary supervised reflection, not pure emergence from patient interaction alone.\n\nWhat is new and solid: a comorbidity / long-tail synthetic patient pipeline (583 diseases, 467 comorbidities), four explicit patient-construction pillars with some diversity and consistency numbers, dual medical-plus-efficiency metrics, and a closed-loop Agent-Kernel baseline that shows the trajectories and a couple of consultation case studies. They ship a GitHub link, keep everything synthetic for privacy, and the limitations section is honest about unimodal text and missing temporal disease progression. Prior hospital sims (Agent Hospital, Agent Clinic, MedAgentSim) already occupy the space; this is a legitimate extension with more evolution tracking and harder cases, not a clean break.\n\nSoft spots in proportion: no ablation that removes or degrades the ground-truth reflection signal, no single-agent or non-consulting controls, no error bars, and heavy LLM-as-judge (GPT-5.2) for both patient validation and treatment alignment. The Kantian “thing-in-itself” framing is decorative. Circularity is real—the same oracle that drives improvement also defines the scores—so the spontaneous-cooperation story is enabled by an action space that already includes consultation rather than proven to arise from interaction alone. None of this makes the engineering useless; it just means the CI language should be tempered.\n\nThis is for people building medical multi-agent sims or evolution arenas who want a concrete, privacy-safe testbed and baseline numbers. It is not a foundation-model data-wall solution and not a clinical tool. I would bring it to reading group as a methods paper, cite the arena if I need a hospital sim, and send it to peer review with a clear request for the missing ablations. The work is serious and reproducible enough to deserve referee time.","headline":"Useful hospital multi-agent arena with real metric trajectories, but the CI / data-in-agent-self story is mostly oracle-guided individual improvement.","tokens_in":11695,"tokens_out":557,"would_cite":true,"duration_ms":10729,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"OpenHospital is a live hospital arena where physician agents evolve collective intelligence by interacting with dynamic patient agents, improving clinical metrics while lowering token cost.","keywords":["collective intelligence","LLM multi-agent systems","medical simulation","agent evolution","data-in-agent-self","clinical benchmarking","synthetic patients"],"falsifier":"Train the same physician agents for the same number of batches using only interaction logs and no ground-truth reflection signal; if examination precision, diagnostic accuracy, treatment alignment, and spontaneous cross-department consultations fail to improve, the central claim is false.","tokens_in":11683,"feed_emoji":"🏥","tokens_out":523,"duration_ms":17349,"temperature":0.7,"pith_summary":"The paper argues that static datasets only capture surface phenomena, so LLM agents stay stuck at imitation and cannot break the data wall. Collective intelligence requires agents to engage dynamic environments that change with their own actions. OpenHospital supplies that environment: physician agents must actively elicit information from patient agents built for clinical correctness, persona diversity, linguistic fluency, and behavioral realism. A closed-loop reflection process after each case lets the physicians accumulate experience. Across successive batches the agents raise examination precision, diagnostic accuracy, and treatment alignment while reducing total input tokens, and they spontaneously begin consulting one another on complex cases. A sympathetic reader cares because the arena both trains multi-agent systems and supplies objective metrics for medical skill and system efficiency, showing that continuous capability growth is possible without new human corpora.","feed_headline":"Hospital agents get smarter and cheaper with each patient","feed_subtitle":"A live doctor-patient arena raises diagnosis scores while cutting tokens and sparking consultation.","key_machinery":"The data-in-agent-self paradigm: physician agents receive no static case files; they must interact with patient agents (treated as dynamic entities) to obtain clinical information, forcing knowledge integration, multi-agent debate, and measurable evolution tracked by examination, diagnosis, treatment, and token metrics.","core_discovery":"As physician agents process successive batches of cases inside OpenHospital under a closed-loop reflection mechanism, their Examination Precision, Diagnostic Accuracy, and Treatment Plan Alignment all rise while total input tokens fall, and cooperative behaviors such as peer consultation emerge spontaneously; the arena therefore both evolves and quantifies LLM-based collective intelligence.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Physician agents raise exam precision and cut tokens with each case batch","OpenHospital evolves doctor agents: higher accuracy, lower tokens, peer talks","Successive patients lift diagnosis scores while agent tokens fall","Closed-loop cases boost medical metrics and spark spontaneous consultation","LLM hospital teams improve accuracy and efficiency under reflection"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The evolution claim rests on the premise that post-case self-critique against ground-truth diagnoses plus synthetic patients validated only by other language models produces genuine collective intelligence rather than guided individual improvement.","fun_headline_variants_meta":{"raw":{"variants":["Physician agents raise exam precision and cut tokens with each case batch","OpenHospital evolves doctor agents: higher accuracy, lower tokens, peer talks","Successive patients lift diagnosis scores while agent tokens fall","Closed-loop cases boost medical metrics and spark spontaneous consultation","LLM hospital teams improve accuracy and efficiency under reflection"]},"model":"grok-4.5","effort":"low","cost_usd":0.006476,"raw_usage":{"total_tokens":1562,"prompt_tokens":628,"num_sources_used":0,"completion_tokens":84,"cost_in_usd_ticks":64760000,"prompt_tokens_details":{"text_tokens":628,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":850,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":628,"tokens_out":84,"duration_ms":7294,"temperature":1.0,"reasoning_tokens":850,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T20:58:55.207403+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train the same physician agents for the same number of batches using only interaction logs and no ground-truth reflection signal; if examination precision, diagnostic accuracy, treatment alignment, and spontaneous cross-department consultations fail to improve, the central claim is false.","supporting_citations":[],"review_version":1}