{"id":"4d2651ff-1dc2-40ee-b96b-680f37cf8ded","arxiv_id":"2501.16215","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"ConMIL uses per-class interpretable multiple instance learning and conformal prediction to feed LLMs calibrated, highlighted hints, lifting accuracy on ECG and EEG visual inspection from 13 to 48 percent to 71 to 97 percent.","lead":"A new decision-support framework, ConMIL, combines a small interpretable model that highlights which parts of an ECG or EEG matter and how confident it is, with a large language model that makes the final call. On two public medical datasets, giving the LLM these hints raised its diagnostic accuracy from 13 to 48 percent standalone to 71 to 97 percent on confident and uncertain samples.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-class FNR guarantee is contradicted by Table A2 (SleepEDF class 5: 10.68% at α=0.05, PTB-XL class 2: 6.31%) and not derived from the cited scalar-λ CRC theorem; this undermines ConMIL's central statistical-reliability claim.","rationale":"The paper's main empirical claim that ConMIL-supported LLMs outperform LLMs alone is broadly supported by the reported stratified accuracies (e.g., 94.92% vs 46.13% for Qwen2-VL-7B on PTB-XL confident samples), and the results are large enough that the core engineering contribution is plausible even with single-run LLM evaluations. However, the distinctive novelty is not just accuracy but valid per-class uncertainty quantification: ConMIL is framed as the first combination of conformal prediction with MIL that provides per-class statistical coverage guarantees. The per-class FNR column of Table A2 is the direct evaluation of that guarantee, and it shows violations at the nominal α=0.05 level. The formal argument is also incomplete: Definition 1 and Theorem 3 are scalar-λ CRC results, while Eq. 14 requires K thresholds; the paper does not prove the risk-monotonicity, boundedness, or finite-sample conditions for the vector-per-class construction. This does not warrant rejection because the empirical framework may still be useful and the issues are correctable, but it does mean the statistical-reliability claim should not be accepted as stated. Thus the existing CONDITIONAL verdict remains appropriate.","tokens_in":33966,"tokens_out":7031,"duration_ms":77745,"concrete_test":"Re-run the released ConMIL code on PTB-XL and SleepEDF with the same subject-wise splits, but implement per-class CRC explicitly: for each class k, use only calibration samples of that class in Eq. 16 and compute test FNR_k = mean(1 - I(y_k ∈ S)). Repeat over 100 random subject-wise calibration/test re-splits and report the distribution of test FNR_k at α=0.05 for PTB-XL class 2 and SleepEDF class 5. If the upper 95% confidence interval of the mean excludes 5%, the per-class guarantee is not valid as stated; if the interval includes 5%, the Table A2 violations are finite-sample noise. Analytically, also verify whether the vector-threshold construction from Eq. 14 satisfies the monotonicity and bounded-risk conditions required to invoke Theorem 3 for a scalar λ; if not, the theorem cannot be applied as written.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing weak point is the claimed per-class statistical guarantee. In Section 3.3, Eq. 12 defines the per-class FNR, Eq. 14 constructs the prediction set from K class-specific thresholds λ_k, and Eqs. 15–16 invoke CRC to select a threshold, with Theorem 3 giving E[FNR] ≤ α under exchangeability. But the quoted CRC theorem is for a scalar λ with a monotone bounded risk function; no proof is supplied that applying it to K thresholds, or to class-conditioned calibration subsets, yields per-class control. The paper's own Table A2 contradicts the guarantee empirically: at α=0.05, PTB-XL class 2 has FNR 6.31±0.65% and SleepEDF class 5 has FNR 10.68±0.71%, both above 5% by more than finite-sample slack (class 5 is a few percent of the SleepEDF calibration data, so the B/(n+1) correction is small). Since the abstract and Section 3.3 sell ConMIL precisely as producing set-valued outputs with per-class statistical coverage guarantees, this internal inconsistency directly undermines the central reliability claim. The LLM-accuracy improvement itself may survive a fix, but the paper would then be an empirical uncertainty-aware decision-support framework, not a statistically guaranteed one.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ConMIL, a decision-support framework that combines multiple instance learning with per-class interpretability (QTrans-Pooling), conformal risk control for set-valued predictions, and prompting of multimodal LLMs for visual inspection of physiological signals. The authors evaluate ConMIL on SleepEDF and PTB-XL using Qwen2-VL-7B, MiMo-VL-7B-RL, and ChatGPT-4.0, reporting large accuracy gains on confident and uncertain samples relative to standalone LLM use, and they provide qualitative case studies and an 'interview' protocol. The main advertised contribution is twofold: per-class interpretable MIL pooling, and a conformalized MIL pipeline that gives per-class false-negative-rate guarantees under exchangeability.","tokens_in":34230,"tokens_out":4088,"duration_ms":42840,"significance":"If the claims hold, the framework is a valuable practical contribution: it combines the interpretability and uncertainty quantification of a small specialized model with the reasoning of a frozen multimodal LLM, without fine-tuning the LLM, and it releases code and uses public datasets. The empirical gains reported are large and the qualitative material is detailed. However, the central statistical-reliability claim is not supported by the paper's own numbers: Table A2 reports per-class FNRs above the promised α at α=0.05, and the proof of per-class control is not supplied. The LLM evaluation also has reproducibility and confound issues. The overall direction is promising and the empirical pipeline is clearly described, but the reliability guarantee needs repair or substantial qualification.","major_comments":[{"comment":"The headline guarantee is per-class coverage: P(y_test in S_α(X_test) | y_test = k) ≥ 1-α for every k. Table A2 directly contradicts this at α=0.05: PTB-XL class 2 has per-class FNR 6.31±0.65% and SleepEDF class 5 has FNR 10.68±0.71%, both above 5% by amounts far exceeding the finite-sample slack B/(n+1). The paper reports marginal coverage (94.65% and 95.59%) that meets α, but marginal coverage is not the advertised per-class guarantee. Since the abstract and Section 3.3 sell the framework precisely as providing per-class statistical guarantees, this internal inconsistency undermines the central reliability claim and must be fixed before publication.","section":"Section 3.3, Eq. (15)-(16), Theorem 3, and Table A2"},{"comment":"The Conformal Risk Control theorem quoted from [38] applies to a single scalar threshold λ with a bounded, right-continuous, non-increasing risk function. Here the prediction set is built from K class-specific thresholds λ_k and the risk in Eq. (12) is class-conditioned. The manuscript does not prove that CRC applied to K thresholds, or to class-conditioned calibration subsets, yields per-class FNR control. The empirical violation in Table A2 suggests this is not just a missing proof but a real gap. The authors should either provide a class-wise calibration argument with explicit finite-sample bounds, or replace the claim with a marginal-coverage guarantee and state that per-class control is not guaranteed.","section":"Section 3.3, Eq. (14) and Definition 1"},{"comment":"The LLM evaluation is confounded by the task asymmetry. In the standalone condition the LLM performs an open 5-way classification; in the ConMIL condition it receives a singleton or a very small set (e.g., two classes) plus per-class heatmaps and is instructed to choose 'without additional analysis.' The large accuracy jump on confident samples may therefore largely reflect ConMIL's own accuracy and the reduced label space, rather than the LLM's visual inspection ability. To support the claim that ConMIL 'enhances visual inspection capabilities,' the authors should report per-sample comparisons, including cases where the ConMIL set contains the true label but the singleton/uncertain split is controlled, and should report the ConMIL-only accuracy on the same confident/uncertain subsets. Without this, the contribution to LLM reasoning, as opposed to providing a nearly solved subset, is not established.","section":"Section 5.3, Figures 3-4 and decision rules in Section 4.2"},{"comment":"All LLM results are single-run, and ChatGPT-4.0 is accessed through the OpenAI GUI, which is not reproducible and may change over time. For a central empirical claim of 94.92% vs. 46.13%, the paper should provide confidence intervals, multiple seeds or repeated API calls, temperature settings, and the exact model versions. The qualitative 30-case study in Table 1 is useful but too small to carry the quantitative weight of the comparison.","section":"Section 5.2 and Section 5.3"}],"minor_comments":[{"comment":"The phrase 'distinctively synergizes' is ungrammatical and should be revised.","section":"Abstract"},{"comment":"The sum in the FNR definition runs from i=0 to N_k; it should run from i=1 to N_k.","section":"Section 3.3, Eq. (12)"},{"comment":"The notation is confusing: Eq. (13) defines a binary prediction ŷ_i using a threshold λ, while Eq. (14) reuses λ as a vector of K thresholds and writes p(X_i) where the per-class probability p_k(X_i) is intended. Please clarify the notation and distinguish scalar and vector thresholds.","section":"Section 3.3, Eq. (13) and Eq. (14)"},{"comment":"Theorem 2 is essentially the elementary fact that conditioning cannot increase entropy. Calling this a reduction in class-wise variability that 'simplifies decision boundaries' overstates what the inequality establishes; please either provide a more substantive statement or present it as a heuristic remark.","section":"Section 3.2, Theorem 2"},{"comment":"There is a typo: 'MModels like Med-PaLM' should be 'Models like Med-PaLM.'","section":"Section 2.4"},{"comment":"The phrase 'per-class interpretabilityg' contains a stray 'g' and needs correction.","section":"Section 2.3"},{"comment":"The decision rule for singleton predictions says the LLM 'interprets the result directly,' but the paper does not specify whether the LLM is allowed to disagree with a singleton ConMIL prediction. This matters for interpreting the confident-sample accuracy numbers.","section":"Section 4.2 and Figure 2"},{"comment":"The majority-voting rule y_i = argmax_k Σ_t y_{i,k} is formally undefined for ties and is not used in the later conformal set construction; please reconcile this with the set-valued prediction definition.","section":"Section 3.1, Eq. (2)"}],"recommendation":"major_revision","confidential_remarks":"The empirical accuracy gains are large and the code release is a strength, but the per-class coverage guarantee is contradicted by the paper's own Table A2. This is a load-bearing issue because the abstract and Section 3.3 advertise statistical per-class guarantees. I would not reject the paper: the framework and the LLM-synergy idea are likely salvageable, and the contradiction may be fixable by either supplying a correct class-wise calibration proof or downgrading the guarantee to marginal coverage with explicit caveats. However, the LLM evaluation also needs to address the confound between reduced label sets and genuine LLM reasoning. If the authors can repair the guarantee and clarify the evaluation, the paper could be a useful contribution to clinical AI decision support."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: read this for the empirical hybrid-pipeline result, not for the coverage theorem. ConMIL delivers large, plausible gains when a small MIL model with per-class attention maps and conformal set-valued outputs is used to prompt frozen VLMs on ECG/EEG. That is a genuinely useful contribution, and the authors deserve credit for shipping code, using public data, and doing controlled comparisons against MILLET and Monty Hall prompting.\n\nWhat is new: replacing TimeMIL's single class token with K class tokens and cross-attention gives per-class interpretability; that is a small but real architectural delta. The integration of conformal prediction with MIL and then feeding the set-valued predictions plus heatmaps to an LLM is new as far as I know. The empirical numbers are dramatic, and the case studies are convincing that the visual prompts, not just the narrowed label set, are responsible.\n\nSoft spots: the headline per-class FNR guarantee does not survive contact with Table A2. At alpha=0.05, PTB-XL class 2 FNR is 6.31% and SleepEDF class 5 FNR is 10.68%, both above the claimed bound. The CRC theorem quoted is for scalar lambda and a monotone risk function; no proof shows that applying it to K independent class thresholds or class-conditioned calibration subsets gives per-class control. This is the load-bearing flaw. Marginal coverage is fine, but the paper sells per-class guarantees. Also, Theorem 2 is just conditional entropy; conditioning on more variables cannot increase entropy, so it is not a real theoretical result. And the LLM evaluations are single-run, with ChatGPT used through the GUI, so sampling variability in the central numbers is unquantified. These are fixable: re-derive or re-calibrate per-class thresholds, report multiple runs, and soften the guarantee language.\n\nWho this is for: people building uncertainty-aware, no-fine-tune LLM decision support for waveform data. It is a solid empirical demonstration, but the statistical-reliability claim needs correction before I would trust it in a clinical setting.\n\nRecommendation: send to peer review. A serious referee can sort out the coverage issue; desk rejection would waste a useful empirical study. My own verdict is conditional acceptance after major revision.","headline":"Solid empirical hybrid-SSM/LLM pipeline; the per-class conformal guarantee is contradicted by the paper's own Table A2, so treat the reliability claim as unproven.","tokens_in":34847,"tokens_out":2160,"would_cite":true,"duration_ms":22447,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ConMIL turns a small conformalized MIL model into a plug-in that lifts frozen LLMs' ECG/EEG reading from 13-48% to ~95% on confident samples.","keywords":["physiological signal classification","multiple instance learning","conformal prediction","large language models","visual inspection","ECG arrhythmia detection","sleep stage classification","interpretability"],"falsifier":"On a fresh exchangeable split of PTB-XL or SleepEDF with alpha=0.05, compute per-class false-negative rates from ConMIL's set-valued predictions; if any class exceeds 5%, as the paper's own supplementary table shows for some classes, the per-class coverage guarantee is not demonstrated.","tokens_in":33739,"feed_emoji":"🩺","tokens_out":8594,"duration_ms":76643,"temperature":0.7,"pith_summary":"The paper sets out to show that the weak spot of large language models on medical waveforms—domain-specific precision—can be fixed without fine-tuning the LLM, by feeding it calibrated hints from a small specialized model. Its ConMIL framework trains a multiple-instance model to say which time segments support each candidate class, then conformalizes those scores so the output is a set of diagnoses with a per-class coverage guarantee. On arrhythmia and sleep-stage data, frozen Qwen2-VL-7B and MiMo-VL-7B-RL supported by ConMIL reach 94.92% and 96.82% precision on confident (singleton) samples, compared with 46.13% and 13.16% when the LLM reads the plot alone. If this holds, it gives a recipe for trustworthy visual inspection of physiological signals that keeps the small model's specialization and the LLM's reasoning, with ambiguous cases explicitly flagged for human review.","feed_headline":"Small-model hints lift LLM ECG/EEG accuracy to ~95%","feed_subtitle":"Conformalized MIL adds per-class heatmaps and calibrated uncertainty; LLMs jump from 13-46% to ~95% on confident cases.","key_machinery":"The load-bearing object is QTrans-Pooling, a pooling layer that gives each of the K classes its own learnable token and computes class-specific attention over time instances through sigmoid cross-attention with bias -log(T). This produces per-class heatmaps that say which time segments support each possible diagnosis. On top of it, conformal risk control calibrates per-class thresholds on a held-out set so that the set-valued prediction contains the true class with probability at least 1-alpha for each class; the decision rules then map singleton, multi-label, and trivial sets to direct acceptance, LLM adjudication, or clinician referral.","core_discovery":"ConMIL is a decision-support framework that combines three things: a per-class interpretable MIL pooling layer called QTrans-Pooling, conformal risk control for set-valued predictions with a per-class false-negative guarantee, and a prompting scheme that hands both the prediction set and per-class heatmaps to a frozen multimodal LLM. The paper's central discovery is that this combination converts an LLM's visual inspection of ECG and EEG from a mostly unreliable guess into a stratified, evidence-based decision: singleton sets are read with high confidence, multi-label sets are adjudicated using per-class interpretations, and trivial sets that include all classes are sent to a clinician. Empirically, this support lifts Qwen2-VL-7B from 46.13% accuracy on PTB-XL and 13.16% on SleepEDF to 94.92% and 96.82% on confident samples, with 70.61%/78.10% and 78.02%/71.98% on uncertain samples, and the same pattern holds for MiMo-VL-7B-RL. ConMIL is described as the first combination of conformal prediction with MIL, and the per-class coverage guarantee is the formal basis for the reliability claims.","pith_inferences":["If ConMIL generalizes, the same plug-in design could be applied to other waveform-reading tasks, such as fetal heart-rate monitoring or intraoperative EEG, whenever a small model can output per-class saliency; the test would be whether the heatmaps remain discriminative for classes that are not mutually exclusive.","A natural deployment is a triage protocol where singleton outputs auto-approve, multi-label outputs go to LLM-assisted review, and trivial outputs go straight to a clinician; the paper's numbers imply most SleepEDF cases are singleton at alpha=0.05, so workload savings could be large.","If per-class heatmaps are the active ingredient, a controlled ablation that keeps the set-valued output but removes the heatmaps would quantify how much of the gain comes from interpretability rather than from mere narrowing of the label set.","Under distribution shift, the non-exchangeable bound in the paper means coverage degrades with the total variation distance between calibration and deployment data; in practice this points to periodic local recalibration before using ConMIL in a new hospital or device population."],"forward_implications":["Both Qwen2-VL-7B and MiMo-VL-7B-RL reach 94.92% precision on confident PTB-XL samples and 96.82% on confident SleepEDF samples, against 46.13%/48.16% and 13.16%/21.08% without support.","On uncertain multi-label samples the same LLMs score 70.61%/78.10% on PTB-XL and 78.02%/71.98% on SleepEDF, still well above their standalone baselines.","Instead of forcing a guess, ConMIL can defer: at alpha=0.05, 171 PTB-XL samples are rejected for clinician review, while SleepEDF produces no trivial sets.","Because the LLM is used frozen, the framework works with proprietary models that cannot be fine-tuned, avoiding the main barrier to domain adaptation.","Per-class heatmaps are what let the LLM separate visually similar classes such as N2 versus REM or ST/T change versus hypertrophy; prediction sets alone, as in the Monty Hall comparison, perform far worse."],"supporting_citations":[{"why":"Supplies the base MIL architecture (InceptionTime features plus wavelet positional encoding and Trans-Pooling) that QTrans-Pooling modifies.","marker":"[5]"},{"why":"Baseline MILLET method whose single-prediction interpretation is compared against ConMIL's per-class heatmaps.","marker":"[6]"},{"why":"Provides Conformal Risk Control, the threshold-selection rule that underwrites the per-class false-negative guarantee.","marker":"[38]"},{"why":"Monty Hall method baseline showing that a conformal prediction set alone improves LLMs far less than ConMIL's full support.","marker":"[32]"},{"why":"Qwen2-VL-7B is one of the frozen multimodal LLMs evaluated with and without ConMIL support.","marker":"[44]"},{"why":"MiMo-VL-7B-RL is the second frozen multimodal LLM used to confirm that ConMIL's benefit is not model-specific.","marker":"[45]"},{"why":"PTB-XL is the arrhythmia dataset used for training, calibration, and testing.","marker":"[41]"},{"why":"SleepEDF is the sleep-staging dataset used for training, calibration, and testing.","marker":"[40]"},{"why":"InceptionTime is the feature extractor inside the MIL model that produces instance embeddings for QTrans-Pooling.","marker":"[3]"}],"fun_headline_variants":["Conformal MIL boosts LLM ECG/EEG accuracy to 95%","LLMs get 95% ECG accuracy with conformalized MIL","Small models add guardrails, lift LLM ECG reads to ~95%","ECG/EEG: conformal MIL makes LLMs reliable"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reliability guarantee depends on the calibration and test data being exchangeable and on per-class thresholds actually capping each class's miss rate at the chosen level.","fun_headline_variants_meta":{"raw":{"variants":["Conformal MIL boosts LLM ECG/EEG accuracy to 95%","LLMs get 95% ECG accuracy with conformalized MIL","Small models add guardrails, lift LLM ECG reads to ~95%","ECG/EEG: conformal MIL makes LLMs reliable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000539,"raw_usage":{"total_tokens":2688,"prompt_tokens":1147,"completion_tokens":1541,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":763,"completion_tokens_details":{"reasoning_tokens":1462}},"tokens_in":763,"tokens_out":1541,"duration_ms":10755,"temperature":1.0,"reasoning_tokens":1462,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:36:57.339856+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a fresh exchangeable split of PTB-XL or SleepEDF with alpha=0.05, compute per-class false-negative rates from ConMIL's set-valued predictions; if any class exceeds 5%, as the paper's own supplementary table shows for some classes, the per-class coverage guarantee is not demonstrated.","supporting_citations":[{"cited_title":"TimeMIL: Advancing multivariate time series classification via a time-aware multiple instance learning","cited_arxiv_id":null,"evidence_quote":"Supplies the base MIL architecture (InceptionTime features plus wavelet positional encoding and Trans-Pooling) that QTrans-Pooling modifies."},{"cited_title":"Inherently interpretable time series classification via multiple instance learning","cited_arxiv_id":null,"evidence_quote":"Baseline MILLET method whose single-prediction interpretation is compared against ConMIL's per-class heatmaps."},{"cited_title":"Mimo-vl technical report, 2025","cited_arxiv_id":null,"evidence_quote":"MiMo-VL-7B-RL is the second frozen multimodal LLM used to confirm that ConMIL's benefit is not model-specific."},{"cited_title":"Ptb-xl, a large publicly available electrocardiography dataset.Scientific data, 7(1):1–15, 2020","cited_arxiv_id":null,"evidence_quote":"PTB-XL is the arrhythmia dataset used for training, calibration, and testing."},{"cited_title":"Analysis of a sleep-dependent neuronal feedback loop: the slow-wave microcontinuity of the eeg.IEEE Transactions on Biomedical Engineering, 47(9):1185–1194, 2000","cited_arxiv_id":null,"evidence_quote":"SleepEDF is the sleep-staging dataset used for training, calibration, and testing."},{"cited_title":"Inceptiontime: Finding alexnet for time series classification.Data Mining and Knowledge Discovery, 34(6):1936–1962, 2020","cited_arxiv_id":null,"evidence_quote":"InceptionTime is the feature extractor inside the MIL model that produces instance embeddings for QTrans-Pooling."}],"review_version":1}