{"id":"48c61932-6163-4239-a90e-c9f890864b47","arxiv_id":"2507.22205","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"CTG-Insight reports 96.4% accuracy on a 50-sample subset of the NeuroFetalNet test set, but the comparison against deep learning baselines is uneven and the system code and exact prompts are not released.","lead":"CTG-Insight is a multi-agent LLM system that reads fetal heart rate and contraction traces and classifies fetal health, producing step-by-step explanations. The authors report 96.4% accuracy, but the evaluation used only 50 samples and compared against deep learning models on a much larger test set.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'state-of-the-art' claim compares CTG-Insight on 50 randomly sampled traces with deep-learning baselines on the full test set; with n=50 the 2.17-point accuracy gap is within sampling error, so superiority is not established.","rationale":"I read the paper in good faith. The framework is plausible, the prompts are clinically detailed, and the authors are candid about limitations. The single most load-bearing issue is the evaluation mismatch: the headline 'state-of-the-art accuracy' rests on comparing 50 LLM-evaluated traces with deep-learning baselines evaluated on the full test set, with no confidence intervals or significance testing. The paper's own Limitations section explicitly acknowledges that the 50-sample subset may limit generalizability and statistical significance, so the evidence does not support the strongest claim as stated. This is not an internal inconsistency but an unquantified comparison; it is also fixable with a full evaluation. Because the core idea could still be valid with additional evidence, I would keep the conditional verdict rather than reject the paper outright.","tokens_in":8613,"tokens_out":8896,"duration_ms":101980,"concrete_test":"Run CTG-Insight on the complete NeuroFetalNet test set with the original label distribution and the same rendering/label mapping used in Section 4.1; compute accuracy and F1 with Wilson confidence intervals, and compare against NeuroFetalNet on the same instances via McNemar or a permutation test. If the accuracy is not significantly above 94.23% (or drops below it), the abstract's state-of-the-art claim should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 states that LLM methods were evaluated on 50 randomly sampled test instances (25 abnormal, 25 normal), while the deep-learning baselines in Table 1 were trained and tested on the NeuroFetalNet train/test split. Table 1 then presents CTG-Insight 96.40%/97.81% as state-of-the-art (abstract; Section 4.2). The central claim assumes a balanced 50-trace subsample is a valid proxy for the full test set. This is unsupported: with n=50, a single misclassification shifts accuracy by 2 percentage points, and the approximate 95% Wilson interval for 96.4% spans roughly 86.6-99.4%, so the 94.23% NeuroFetalNet result is inside the interval. The paper reports no per-trial counts, standard deviations, or significance test, and the five repeated trials are not five independent samples if they resample from a small pool. The forced 50/50 label balance may also differ from the test-set prevalence, making accuracy/F1 non-comparable. The Limitations section concedes that the 50-sample subset 'may limit generalizability and statistical significance' and defers full-scale evaluation to future work. The interpretability half of the claim also lacks evidence: feature-agent classifications are never compared with expert annotations. Thus the strongest quantitative and qualitative claims are both asserted rather than demonstrated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents CTG-Insight, a multi-agent LLM framework that decomposes cardiotocography (CTG) traces into five clinically defined features—baseline, variability, accelerations, decelerations, and sinusoidal pattern—each handled by a dedicated feature agent, and then uses an aggregator agent to produce a final normal/suspicious/pathological classification with a natural-language explanation. The authors evaluate CTG-Insight on the NeuroFetalNet Dataset and report 96.40% accuracy and 97.81% F1-score, which they compare against deep learning baselines and a single-agent direct-prompt LLM baseline, claiming state-of-the-art performance and interpretability. The appendices include the full system prompts based on FIGO and German S1 guidelines.","tokens_in":8872,"tokens_out":2669,"duration_ms":34009,"significance":"If the performance and interpretability claims were properly supported, the framework would be a useful step toward transparent fetal monitoring: the modular feature-agent design mirrors clinical reasoning, the prompts are grounded in established guidelines, and the use of a public dataset is a strength. However, the current evidence does not substantiate the headline claims. The comparison is uneven because the LLM is tested on 50 balanced samples while deep learning baselines are tested on the full test set; no confidence intervals or significance tests are reported; and the interpretability claim is asserted without any validation of the feature agents' intermediate outputs. The contribution is potentially valuable, but the evaluation needs substantial strengthening before the claims can be accepted.","major_comments":[{"comment":"The central performance claim rests on an invalid comparison. Section 4.1 states that LLM-based methods were evaluated on 50 randomly sampled test instances (25 abnormal, 25 normal), while deep learning baselines in Table 1 were trained and tested on the full NeuroFetalNet train/test split. Table 1 then directly compares CTG-Insight's 96.40%/97.81% with NeuroFetalNet's 94.23%/94.20%. With n=50, a single misclassification changes accuracy by 2 percentage points, and the approximate 95% Wilson interval for 96.4% spans roughly 86.6-99.4%, so the reported gap is not statistically meaningful. The authors must either evaluate all methods on the same test set (or the same 50-trace subset) and report per-trial counts, standard deviations, and significance tests, or withdraw the state-of-the-art claim.","section":"Section 4.1, Table 1"},{"comment":"The Limitations section acknowledges that the 50-sample subset 'may limit generalizability and statistical significance' and defers full-scale evaluation to future work. This directly contradicts the abstract's claim of 'state-of-the-art accuracy (96.4%) and F1-score (97.8%)'. A claim of state-of-the-art performance cannot be made from a small, convenience subsample while explicitly conceding that the evaluation lacks statistical significance. This issue is load-bearing because the entire contribution is framed around this quantitative result.","section":"Section 5 (Limitations)"},{"comment":"The accelerations rule 'Normal: two accelerations in 20minutes' is described in the methodology as a 'slight modification' of clinical guidelines to account for the 20-minute data window, but no justification, sensitivity analysis, or validation is provided. If thresholds were tuned after seeing test labels or after observing the system's errors, this introduces a circularity risk. The authors should justify each adaptation with reference to the original guideline and report how classification changes under alternative thresholds (e.g., one vs. two accelerations per 20 minutes).","section":"Appendix A.4 (Accelerations Analysis Prompt)"},{"comment":"The interpretability half of the claim is not measured. The feature agents output normal/suspicious/pathological classifications and explanations, but these intermediate outputs are never compared against expert annotations or clinician judgments. The aggregator's final explanation is therefore not validated as faithful or clinically sound. Without this, 'transparent and interpretable outputs' is an assertion, not a demonstrated result. The authors should report agreement between each feature agent and expert labels, and ideally human evaluation of explanation quality.","section":"Section 4.2, Table 1"}],"minor_comments":[{"comment":"The paper states that NeuroFetalNet provides binary labels with 'suspicious cases categorized as abnormal', yet the system outputs three categories (normal, suspicious, pathological). The mapping between the three-class output and the binary ground truth used for accuracy/F1 should be clarified.","section":"Section 3.1"},{"comment":"The repeated trials are described only as 'five repeated trials'; it is unclear whether the 50 instances are resampled independently each trial or whether the same 50 instances are used with different LLM sampling. This should be specified, as the effective sample size for the reported averages depends on it.","section":"Section 4.1"},{"comment":"The prompts omit the 'Example Output' entries for space-saving reasons, but the paper claims these ensure consistent formatting. Including at least one full example per agent would improve reproducibility, especially since the LLM's output format directly affects the aggregator's input.","section":"Appendix A.2-A.6"},{"comment":"No details are given about the LLM inference settings (temperature, max tokens, model version beyond 'GPT-4.1', number of runs per trial). These details matter for reproducibility of LLM-based results.","section":"Section 3.2.2"},{"comment":"There are several minor typographical issues, including 'the the' in the ACM reference format line and inconsistent spacing in '20minutes' in Appendix A.4. A careful proofreading pass is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is a short companion paper, and the evaluation section is too thin to support the state-of-the-art claim. I recommend major revision rather than rejection because the core framework is potentially sound and the claims could be made supportable with a properly matched evaluation on the full test set, confidence intervals or significance tests, and validation of the feature-level explanations. The editor may also wish to consider whether the statistical rigor expected for a medical AI claim is appropriate for a 7-page companion paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this is a plausible, well-motivated idea whose central 'state-of-the-art' claim does not survive contact with the evaluation. The authors build a multi-agent LLM system that decomposes CTG traces into five clinically defined features, guided by FIGO and S1 guidelines, and an aggregator produces a final classification with explanations. That decomposition is sensible and the prompt design is genuinely grounded in medical guidelines. The formative analysis of existing fetal monitoring apps is a nice touch and speaks to a real HCI need.\n\nThe trouble is Table 1. CTG-Insight is evaluated on 50 randomly sampled test traces (25 normal, 25 abnormal), while the deep learning baselines are evaluated on the full test set. With n=50, one misclassification changes accuracy by two points, and the 96.40% figure has a 95% Wilson interval that easily includes NeuroFetalNet's 94.23%. No standard deviations, per-trial counts, or significance tests are reported. So the claim that CTG-Insight is state-of-the-art is not supported by the evidence. The authors do acknowledge the 50-sample limitation in their Limitations section, which is honest, but the abstract and Section 4.2 nevertheless assert SOTA performance.\n\nThe interpretability half of the claim also needs scrutiny. The feature agents' intermediate classifications are never compared against expert annotations. We don't know whether the explanations are correct or misleading — they may be fluent but wrong. The 'slight modifications' to guideline thresholds (e.g., 'two accelerations in 20 minutes') are not independently justified and could have been tuned to the test set.\n\nThe core idea is sound, and the paper is clearly written and honest about many limitations. But the quantitative and qualitative claims are both asserted rather than demonstrated. A full evaluation on the entire test set, with error bars, a significance test, and at least a small expert annotation of feature-level outputs, is needed before the claims can be taken seriously.\n\nWho this is for: researchers working on LLM-based interpretable systems for clinical signals, and HCI folks interested in patient-facing monitoring. It deserves a serious referee, but that referee should demand substantial revision and additional evidence. My recommendation: treat it as a promising workshop paper, not a verified result.","headline":"Well-motivated multi-agent LLM for CTG interpretation, but the state-of-the-art claim rests on a 50-sample evaluation that does not support it.","tokens_in":9411,"tokens_out":1860,"would_cite":false,"duration_ms":18240,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CTG-Insight claims 96.4% accuracy on fetal heart traces by splitting analysis into five guideline-based LLM agents.","keywords":["cardiotocography","large language model","multi-agent system","interpretability","fetal health monitoring","NeuroFetalNet dataset","clinical guidelines","feature decomposition"],"falsifier":"Run CTG-Insight on the full NeuroFetalNet test partition, using the same evaluation protocol as the deep learning baselines, and compare accuracy and F1; if accuracy drops below 94.23% or F1 below 94.20%, the paper's central claim of state-of-the-art accuracy is refuted.","tokens_in":8386,"feed_emoji":"🤰","tokens_out":9443,"duration_ms":88900,"temperature":0.7,"pith_summary":"Cardiotocography (CTG) records fetal heart rate and uterine contractions, but most monitoring apps show raw traces that expectant parents cannot interpret. This paper introduces CTG-Insight, a system that splits the analysis of a CTG trace into five medically defined features—baseline, variability, accelerations, decelerations, and sinusoidal pattern—each judged by its own LLM agent, with a final agent combining their verdicts into a fetal-health classification and a natural-language explanation. On the NeuroFetalNet dataset, the authors report 96.40% accuracy and 97.81% F1-score, surpassing the best deep learning baseline (NeuroFetalNet, 94.23% accuracy) as well as a single-agent LLM baseline (79.80%). The paper's aim is to show that structured, guideline-based decomposition makes an LLM both more accurate and genuinely interpretable, so that users receive not only a label but the medical reasoning behind it.","feed_headline":"Fetal heart traces: LLM agents hit 96.4% accuracy","feed_subtitle":"Breaks each trace into five guideline-based features, giving parents a readable explanation of every verdict.","key_machinery":"The central mechanism is a pipeline of five parallel LLM agents plus an aggregator. Each feature agent is governed by a prompt containing the feature's clinical definition, a normal/suspicious/pathological rule set (from the FIGO and German S1 guidelines, with duration thresholds adapted to the 20-minute traces), the agent's role, and example outputs; the decelerations agent also receives a taxonomy of deceleration types. The aggregator applies the overall CTG classification policy—normal only if all features are normal, suspicious if one feature is suspicious, pathological if any feature is pathological or at least two are suspicious—and produces a final classification with a comprehensive explanation. This decomposition is what prevents the model from missing instructions in long single prompts and is what makes the system's reasoning auditable feature by feature.","core_discovery":"The paper claims that a multi-agent LLM organized around clinical feature definitions can classify fetal health more accurately than end-to-end deep learning models while remaining transparent. Using the NeuroFetalNet dataset, CTG-Insight achieves 96.40% accuracy and a 97.81% F1-score, beating NeuroFetalNet (94.23%/94.20%), CNN+BiGRU (84.04%/84.16%), ResNet (82.88%/82.79%), and the Direct Prompt baseline (79.80%/80.10%). The authors attribute the improvement to modular decomposition, which avoids the instruction-following failures that occur when a single LLM is given a very long prompt, and to the explicit encoding of FIGO and S1 clinical guidelines into each agent's rule set. They argue that interpretability is built in, because every feature agent's classification and explanation can be inspected individually before the aggregator makes a final judgment.","pith_inferences":["A decisive test of the claim is a full-test-set run: if CTG-Insight's accuracy falls below the 94.23% reported for NeuroFetalNet when evaluated on the entire test partition, the 'state-of-the-art' comparison would not survive.","The feature agents' intermediate classifications are never checked against expert annotations, so the explanation quality is currently assumed; a study comparing each agent's output with clinician judgments would show whether the transparency is real or merely plausible text.","The paper's 'slight modifications' to clinical guidelines for the 20-minute traces raise a general concern: any LLM that embodies a clinical rule set needs a formal audit that the rules are the ones clinicians actually use at the intended trace length.","The five-agent-plus-aggregator template is naturally transferable to other guideline-driven signal monitoring tasks, such as ECG or EEG triage, where a small set of named features captures most diagnostic variance."],"forward_implications":["If the reported accuracy holds on a full-scale evaluation, LLM-based systems could match or exceed specialized deep learning classifiers on CTG while also supplying feature-level rationales for each decision.","The modular architecture is extensible: adding a new signal feature or updating a guideline threshold requires editing only one agent's prompt, not retraining a whole model.","Because every feature agent's verdict is emitted separately, clinicians can audit which feature drove a suspicious or pathological classification, supporting transparent triage in remote monitoring.","The large gap between CTG-Insight and Direct Prompt suggests that task decomposition into guideline-defined subtasks materially improves LLM performance on physiological signal interpretation."],"supporting_citations":[{"why":"Supplies the NeuroFetalNet dataset and the strongest deep learning baseline (94.23% accuracy) that CTG-Insight claims to beat.","marker":"[30]"},{"why":"Provides the CNN+BiGRU baseline achieving 84.04% accuracy.","marker":"[18]"},{"why":"Supplies the ResNet architecture used as a deep learning baseline (82.88% accuracy).","marker":"[12]"},{"why":"FIGO consensus guidelines, the primary source of the classification rules encoded in the feature-agent prompts.","marker":"[3]"},{"why":"German S1-guideline, the second clinical source for the feature definitions and rule sets.","marker":"[25]"},{"why":"Supports the paper's explanation that long prompts make LLMs forget instructions, motivating the multi-agent decomposition.","marker":"[1]"},{"why":"Also supports the prompt-length effect on LLM performance, used to explain Direct Prompt's lower accuracy.","marker":"[16]"}],"fun_headline_variants":["Multi-agent LLM reads fetal heart traces at 96.4% accuracy","LLM agents explain fetal heart traces, hit 96.4% accuracy","Interpretable LLM for fetal heart traces achieves 96.4% accuracy","Multi-agent AI decodes CTG signals with 96.4% accuracy","LLM framework turns fetal heart traces into clear verdicts at 96.4%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that CTG-Insight outperforms deep learning baselines rests on evaluating it on a randomly sampled balanced subset of 50 test traces, while the deep learning baselines were scored on the full test set, and on clinical rule adaptations that are not clinically validated.","fun_headline_variants_meta":{"raw":{"variants":["Multi-agent LLM reads fetal heart traces at 96.4% accuracy","LLM agents explain fetal heart traces, hit 96.4% accuracy","Interpretable LLM for fetal heart traces achieves 96.4% accuracy","Multi-agent AI decodes CTG signals with 96.4% accuracy","LLM framework turns fetal heart traces into clear verdicts at 96.4%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000705,"raw_usage":{"total_tokens":3178,"prompt_tokens":941,"completion_tokens":2237,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":2134}},"tokens_in":557,"tokens_out":2237,"duration_ms":17429,"temperature":1.0,"reasoning_tokens":2134,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:56:25.479069+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CTG-Insight on the full NeuroFetalNet test partition, using the same evaluation protocol as the deep learning baselines, and compare accuracy and F1; if accuracy drops below 94.23% or F1 below 94.20%, the paper's central claim of state-of-the-art accuracy is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the NeuroFetalNet dataset and the strongest deep learning baseline (94.23% accuracy) that CTG-Insight claims to beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CNN+BiGRU baseline achieving 84.04% accuracy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FIGO consensus guidelines, the primary source of the classification rules encoded in the feature-agent prompts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"German S1-guideline, the second clinical source for the feature definitions and rule sets."},{"cited_title":"Can't Remember Details in Long Documents? You Need Some R&R","cited_arxiv_id":"2403.05004","evidence_quote":"Supports the paper's explanation that long prompts make LLMs forget instructions, motivating the multi-agent decomposition."}],"review_version":1}