{"id":"48657fa4-32f9-4c27-b270-d595c0198584","arxiv_id":"2607.25340","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A multi-agent system spans arrhythmia detection to patient-level management decisions, citing clinical guidelines, and scores highest on every evaluated axis against general vision-language models.","lead":"Cardiologent is a multi-agent AI system that reads a hospital patient's ECG and pulse signal, identifies arrhythmias across a whole record, and turns them into an urgency and treatment recommendation tied to cited clinical guidelines. In tests on operative monitoring data it ranked first against five general AI models on diagnosis, urgency, and management decisions, judged by an AI and by two cardiologists.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"P3 management rubric's 'correct citation' anchor rewards Cardiologent's guideline retrieval, which baselines lack; the 20-patient cardiologist validation is too small to establish the judge as a clinical proxy, so 'scores highest on every axis' may reflect rubric alignment rather than clinical decis","rationale":"The reader's weakest assumption identifies precisely the load-bearing issue: the evaluation instrument (LLM-judge rubric plus small human validation) may be aligned with Cardiologent's engineered behavior—guideline retrieval and citation enforcement—rather than with real clinical decision quality. I agree with this assessment. The rubrics are not merely a neutral measurement tool; the P3 anchor explicitly rewards correct citations, and only Cardiologent has a retrieval mechanism designed to produce them. This makes the comparative claim on P3, and possibly the overall ordering, dependent on an asymmetry in access to external guidelines rather than on clinical reasoning alone. The paper does have independent support: bootstrap CIs, blinded cardiologist ordering, and a hard-label diagnosis accuracy check (Table 11) that does not depend on the rubric. But those do not eliminate the concern that the P3 margin, which is large, may be inflated by citation availability. The proposed redaction test is concrete, low-cost, and would settle whether the headline superiority transfers to actual clinical decision quality. Since the internal evaluation is otherwise sound and the concern is about external validity, the verdict should remain CONDITIONAL pending that test.","tokens_in":29131,"tokens_out":6615,"duration_ms":72243,"concrete_test":"Redact all in-text guideline citations and PMID tags from Cardiologent's patient-level P1–P3 outputs (leaving the clinical content intact) and rescore the full 65-patient set with the same DeepSeek-V4-Pro judge and the original rubric. If the P3 margin over GPT-4o (4.55 vs. 2.94 = 1.61) or the aggregate margin (13.6 vs. 8.6 = 5.0) collapses to within the bootstrap 95% CI of zero, the 'scores highest on every axis' claim is an artifact of citation availability, not clinical decision quality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim 'scores highest on every axis' rests on an evaluation instrument that is partly constructed from the system's own design. The patient-level P3 rubric awards grade 4 for management that is 'specific to the rhythm and patient, guideline-concordant, with a correct citation'; grade 3 penalizes a 'loose' citation. Cardiologent is explicitly engineered to satisfy this: RAG retrieves guideline passages with embedded PMIDs, and the critic enforces that every management claim cites a retrieved PMID. Baselines receive only raw signals, plots, and metadata—they are given no guideline corpus and no retrieval tool, so a correct citation is effectively impossible for them. The P3 comparison therefore measures access to a knowledge source as much as clinical decision quality. The human validation (20 patients, two cardiologists) is too small to establish that the judge's ordering reflects clinical quality rather than citation presence, and the cardiologists themselves may be influenced by authoritative-looking citations. The paper's own Limitations note that the evaluation is retrospective and does not measure patient outcomes, so the 'clinical decision' scores remain a proxy. If the rubric systematically rewards citation availability, the headline superiority over baselines is an artifact of the retrieval-and-citation mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper defines patient-level arrhythmia decision support as a task spanning window-level classification (single-lead ECG and PPG) through patient-level integrated diagnosis, clinical significance, and urgency/management. Cardiologent comprises per-signal LLaVA-7B specialists augmented with deterministic measurement tools and debate-based fusion; at patient level it aggregates window verdicts into a rhythm profile, retrieves guideline passages with PMIDs, reasons over the profile and metadata, and applies a critic that checks claims against retrieved citations. The authors evaluate window classification (macro-F1), reasoning probes (T1–T3), grounding (IoU/MCC), and patient-level P1–P3 scoring using DeepSeek-V4-Pro as an LLM judge on 65 patients and two blinded cardiologists on a 20-patient subset. They report that Cardiologent ranks first on every axis under both raters, that judge–cardiologist ICC is comparable to human–human agreement, and that window macro-F1 is 0.449 versus 0.255 for the strongest baseline.","tokens_in":29418,"tokens_out":5910,"duration_ms":62844,"significance":"If the evaluation is accepted, the contribution is significant: the system is, to my knowledge, the first to assemble a patient-level arrhythmia assessment from raw ECG+PPG through to a guideline-cited management decision, and it does so with a carefully motivated tool-and-debate architecture. The paper has real strengths: a patient-disjoint split, a 3,000-window evaluation sample, bootstrap confidence intervals, per-axis anchored rubrics, a hard-label diagnosis check that cannot be inflated by the rubric, blinded cardiologist ratings, and a promise of released code/data. The window-level results are credible and the ablation of the agent layer over the specialist is informative. However, the patient-level evaluation, which carries the headline 'scores highest on every axis,' has load-bearing weaknesses in the construction of the P3 rubric, the symmetry of information between system and baselines, and the size of the human validation.","major_comments":[{"comment":"The P3 management rubric awards grade 4 to a plan that is 'specific to the rhythm and patient, guideline-concordant, with a correct citation' and grade 3 penalizes a 'loose' citation. Cardiologent alone is equipped with a curated guideline corpus whose passages carry PMIDs (Patient-Level Pipeline, Retrieval) and a critic that enforces that management claims cite a retrieved PMID (Critic bullet 'Citations'). Baselines receive no guideline text or retrieval tool; their prompt asks only for a JSON P3 grounded in waveform and metadata. Table 4 P3 therefore conflates decision quality with access to the citation mechanism. Since the paper's headline is 'scores highest on every axis,' this is load-bearing. The authors should either give baselines the same retrieval resource, rate under a rubric that does not reward citations, or ablate retrieval/critic to show the margin is not explained by cit","section":"Scoring Rubrics (P3), Patient-Level Pipeline, Baseline Prompts"},{"comment":"The at-scale judge is validated on two cardiologists and twenty patients. The claim that judge–cardiologist agreement (ICC 0.74, 0.66) 'matches' human–human agreement (0.67) is based on 20 patients; per-task confidence intervals are wide, e.g., P3 human–human ICC = 0.480 ± 0.170, Judge↔R1 = 0.518 ± 0.172, Judge↔R2 = 0.488 ± 0.166. This is compatible with the judge being a poor proxy, and the 20-patient subset is too small to rule out that agreement is driven by surface features such as citations. The paper itself states in Limitations: 'The evaluation is also retrospective—we score the decision itself, not whether acting on it would have changed the patient's course.' As the sole clinical validation for a strong comparative claim, this needs substantially more data or appropriately hedged wording.","section":"Table 13, Figure 4, Limitations"},{"comment":"The patient-level anchors are explicitly 'written against clinical practice' using the 2023 AF/CHA2DS2-VASc, 2017 VT/SCD, and 2018 bradycardia guidelines — the same guidelines that constitute the curated retrieval corpus and the same citations the critic enforces. The evaluation instrument is therefore aligned with the system's design: P2 context rewards patient-feature-to-risk reasoning that retrieval is built to supply, and P3 management rewards the guideline-cited plan that retrieval and critic are built to produce. This is not a derivation circularity, but it means the comparison measures closeness to the specific guidelines Cardiologent was engineered to cite. At minimum, the paper should show that baselines fail on these axes even when provided the same guidelines, or that the judge's ordering persists for clinical correctness independent of citation presence.","section":"Scoring Rubrics, 'Where the anchors come from'"}],"minor_comments":[{"comment":"Qwen2.5-VL-72B is cited as '(?)' in the baseline enumeration; this is a missing reference placeholder and should be completed.","section":"Experiments, baseline list"},{"comment":"Typo: 'Around its it deterministic tools' should read 'Around it are deterministic tools.'","section":"Method, ECG and PPG Agents"},{"comment":"The caption says 'Setting both is nine-class (ECG specialist)', but the 'Cardiologent' column in that table is the full fused window system, not the ECG specialist alone; please clarify the column semantics to avoid confusion.","section":"Table 9 caption"},{"comment":"The caption says 'seven on the ECG, nine on the PPG' for tools; the main text says the ECG agent has seven tools and the PPG agent has nine. This is correct but easy to misread as class counts; consider rewording.","section":"Figure 3 caption"}],"recommendation":"major_revision","confidential_remarks":"I recommend major revision rather than rejection. The engineering and window-level evaluation are solid, and the limitations are stated honestly. The key obstacle is the patient-level P3 rubric/baseline asymmetry and the very small human-validation sample; a revision that adds a baseline-with-guidelines or citation-ablated condition, and that tempers the 'every axis' claim to match the evidence, would be credible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is one of the better applied clinical-AI papers I've seen recently: it defines a real task (patient-level decision support, not just rhythm labeling), builds a coherent pipeline to span it, and evaluates on a patient-disjoint split with bootstrap CIs, per-axis rubric breakdowns, and blinded cardiologist scoring. The window-level gains are not a mirage: on rare arrhythmias (MAT, SND, AVB) the baselines are at 0.00–0.01 F1 while Cardiologent holds 0.12–0.57, and the hard label match for the dominant rhythm (exact 0.48 vs 0.17 for GPT-4o) is the kind of result you can trust even when rubrics are suspect.\n\nSecond, the headline superiority claim is softer than it looks. The P3 management rubric gives top marks for a “correct citation” to a specific guideline/PMID. Cardiologent is explicitly engineered to produce that: RAG retrieves guideline passages with embedded PMIDs and the critic rewrites or deletes claims not carrying one. The baselines are given raw signals, plots, and metadata — no guideline corpus, no retrieval tool — so a correct citation is nearly impossible for them. The P3 comparison is measuring access to a knowledge source as much as clinical judgment. The same concern leaks into P2 where the anchors reward using patient context, which the system has been designed to do. The human validation (20 patients, two cardiologists) is too small to establish that the judge's ordering reflects clinical quality rather than citation presence; the ICC agreement is suggestive but the CIs are wide, and the cardiologists' own P3 agreement is only 0.48.\n\nThat said, the window-level results and the hard-label diagnosis accuracy largely hold up. The P1 diagnosis claim does not depend on the rubric: exact dominant-rhythm match is a label comparison. The limitations section is honest: single institution, rare classes thin, PPG-only arm never tested on an actual wearable, retrospective evaluation without outcomes. The citation placeholder (Qwen2.5-VL-72B (?) in the baselines list) should be fixed.\n\nBottom line: this deserves a serious referee. The central task formulation is new, the system is a real engineering contribution, and the evaluation mostly supports the diagnostic claims. What needs work is the claim that the management/urgency scores reflect clinical decision quality; a reviewer should push for a study where the judge rubric is not aligned with the system's design and where cardiologists assess management without citations present, plus a larger human sample. I'd accept it for review, with the expectation of major revision on the evaluation.","headline":"Solid multi-agent pipeline for a genuinely new task, but the 'wins on every axis' claim is partly an artifact of an evaluation rubric that rewards the guideline-citation machinery the system itself contains.","tokens_in":29943,"tokens_out":2551,"would_cite":true,"duration_ms":26916,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that the clinical act for arrhythmias lies between naming a rhythm and deciding what to do, and presents a multi-agent system that spans that span, from raw ECG and pulse windows to a patient-level rhythm profile and guide","keywords":["patient-level arrhythmia decision support","multi-agent system","ECG and PPG monitoring","rhythm profile assembly","guideline-concordant management","auditable clinical decisions","LLM-as-judge evaluation","wearable arrhythmia surveillance"],"falsifier":"A blinded, multi-centre evaluation in which cardiologists score management decisions on 100+ patients without knowing the system's identity; if Cardiologent is not ranked first, or if swapping the guideline corpus for a different authoritative set changes the ranking, the central claim is refuted.","tokens_in":28988,"feed_emoji":"🫀","tokens_out":7333,"duration_ms":76080,"temperature":0.7,"pith_summary":"This paper tries to establish that patient-level arrhythmia decision support—deciding what a monitored record means for this patient and what to do about it—is a distinct task that can be built end to end, and that its system, Cardiologent, is the first to span it from raw signals to auditable management decisions. The driving observation is that identical arrhythmia signals can warrant opposite decisions depending on the patient, so the system must assemble window-level readings into a patient rhythm profile and read that profile against the patient's own data and clinical guidelines. The authors claim Cardiologent scores highest on every evaluated axis—integrated diagnosis, clinical significance, urgency, and management—under both blinded cardiologists and a large-scale LLM judge, and that its window-level classification macro-F1 (0.449) is roughly double the strongest baseline's. A sympathetic reader would care because earlier systems stop either at naming the rhythm or at interpreting device-supplied findings, leaving out the judgement a clinician would act on.","feed_headline":"Multi-agent system tops cardiologist-scored arrhythmia decisions","feed_subtitle":"It builds a patient's rhythm profile from raw ECG and pulse, then ties every recommendation to a cited guideline.","key_machinery":"The central object is Cardiologent's two-level pipeline. The load-bearing identity is the per-feature authority assignment during fusion: rather than giving one sensor fixed precedence, a referee decides which sensor's measured evidence is trustworthy for each feature—the ECG for QRS width and P waves, the PPG for pulse regularity and perfusion—so a correct minority reading can survive. The other load-bearing mechanism is the patient-level critic, which holds every diagnosis, significance claim, and management recommendation to a retrieved and cited guideline passage, deleting or rewriting claims that fail that test. The same analyst–devil's advocate–referee debate primitive appears at both","core_discovery":"At the paper's center is the claim that the clinical act is the span between naming a rhythm and knowing what to do, and that this span can be mechanized. Cardiologent is a two-level multi-agent system: at the window level, separate agents read a single ECG lead and a wearable PPG through deterministic measurement tools—heart rate, R–R irregularity, QRS width, P-wave presence, pulse morphology—so every claim is tied to a measured quantity; the two readings are reconciled by a debate in which analysts, a devil's advocate, and a referee grant authority feature-by-feature to the signal that can see the feature. At the patient level, the window verdicts are aggregated into a rhythm profile and o","pith_inferences":["Inference: If the rubric is indeed measuring decision quality, the two-level design—grounded per-signal reading, feature-level arbitration, citation-checked reasoning—should transfer to other continuously monitored signals and other guideline-driven care decisions; that transfer is not tested in the paper.","Inference: The ranking may be partly an artifact of writing rubric anchors from the same guidelines the system retrieves; swapping in a different authoritative guideline set and re-ranking would show whether the advantage is about following guidelines generally or about matching this particular corpus.","Inference: The evaluation is retrospective and from one institution's monitoring data, so the implicit next claim—that the system improves live clinical decisions—remains untested; a prospective deployment study would be the natural sequel.","Inference: The debate mechanism suggests a more general principle: when two sensors have complementary physical access to a feature, authority should be assigned per feature rather than by a fixed sensor hierarchy."],"forward_implications":["If correct, continuous monitoring no longer needs a separate device that pre-detects arrhythmias: the same system can read raw ECG and pulse windows, assemble the patient's rhythm profile, and return a management recommendation.","A wearable could run the pulse-only agent alone for low-cost surveillance and escalate to a combined ECG-plus-pulse read when the pulse reading warrants it, without changing systems.","The reported advantage concentrates on rare, high-risk rhythms—ventricular tachycardia, AV block, sinus node dysfunction—where general models read near zero; this is where a deployed system would add the most.","Because every conclusion is tied to a retrieved, cited guideline passage, a clinician can audit and overturn the system's decision rather than accept an unexplained alert.","The patient-level results imply that evaluating report fluency would miss the real failure mode: general models are fluent but confidently wrong about the patient, so decision-level scoring is necessary."],"fun_headline_variants":["Multi-agent system tops cardiologist-scored arrhythmia decisions","From ECG to action: AI agent ties every decision to a guideline","Arrhythmia AI that audits itself: each call cites its source","Patient-level arrhythmia decisions, not just rhythm names","Cardiologent: auditable AI for arrhythmia care, beats baselines"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the scoring instrument—an LLM-judge rubric whose grade descriptions are written from the same guidelines the system retrieves, with human validation on only 20 patients and two cardiologists—captures real clinical decision quality. If the scores mostly measure closeness to those specific guidelines, the 'highest on every axis' claim does not transfer to clinical use.","fun_headline_variants_meta":{"raw":{"variants":["Multi-agent system tops cardiologist-scored arrhythmia decisions","From ECG to action: AI agent ties every decision to a guideline","Arrhythmia AI that audits itself: each call cites its source","Patient-level arrhythmia decisions, not just rhythm names","Cardiologent: auditable AI for arrhythmia care, beats baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000262,"raw_usage":{"total_tokens":1493,"prompt_tokens":867,"completion_tokens":626,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":537}},"tokens_in":611,"tokens_out":626,"duration_ms":7451,"temperature":1.0,"reasoning_tokens":537,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T02:42:06.976765+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A blinded, multi-centre evaluation in which cardiologists score management decisions on 100+ patients without knowing the system's identity; if Cardiologent is not ranked first, or if swapping the guideline corpus for a different authoritative set changes the ranking, the central claim is refuted.","supporting_citations":[],"review_version":1}