{"id":"050a6460-5c05-4f3b-8b54-a2358c6ea7b4","arxiv_id":"2502.04372","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A conformal active learning pipeline is claimed to classify unstructured texts using only 200 labeled examples, but the evaluation lacks an independent test set and the conformal guarantee is invalid under active selection.","lead":"The paper proposes a framework that combines conformal prediction with active learning to classify unstructured texts using very few manually labeled examples, and tests it on Amazon product reviews as a stand-in for medical records. A generalist might care because it promises cheaper, privacy-preserving text mining for disease surveillance, but the evidence is weakened by evaluation and validity problems.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported accuracy and AUC-ROC appear to be computed on the actively labeled texts themselves, not on a held-out test set; without an independent test split, the central claim of strong performance from 200 labels is unsupported.","rationale":"The reader's weakest_assumption identifies precisely the same load-bearing concern: the reported metrics appear to be computed on the labeled data rather than on an independent test set, and the Yes/No counts in the results tables match the manual label counts. This is not a stylistic issue; it determines whether the headline empirical result exists at all. A secondary concern is that the conformal coverage guarantee in Theorem 1 is invoked even though the active learning loop adaptively selects points, which breaks the exchangeability assumption, but that concern is less central than the missing test split. The paper does release open-source code and a web interface, which is commendable and makes the proposed check feasible, but those assets do not by themselves supply the missing evaluation. Therefore the reader's REJECT verdict stands, and no verdict adjustment is needed.","tokens_in":8450,"tokens_out":2646,"duration_ms":24915,"concrete_test":"Inspect the released code (gitlab.com/nanogennari/olim-learner) and locate the evaluation path where accuracy and AUC-ROC are computed. Specifically, check whether the evaluation set is disjoint from all texts selected for manual labeling during active learning. If no such disjoint held-out set exists, recompute Table 1 using a randomly held-out 20% of the Amazon reviews that is never touched by the active learning loop. If the held-out AUC-ROC for Pet product drops materially below the reported 0.94, or if the code cannot produce a held-out number without modification, the central claim of strong generalization from 200 labels is not supported by the current evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the framework achieves strong performance with as few as 200 manually labeled texts. The evidence for this is Tables 1, 3, and 4. However, the manuscript never describes a held-out test set disjoint from the data used in the active learning loop. The only split mentioned is \"20% of the labeled data was randomly assigned to the validation dataset in each cycle\" (Experimental Setup, Parameters), and that split is described as serving conformal calibration, not as an independent evaluation set. The Yes/No columns in Tables 1, 3, and 4 sum to exactly the number of manually labeled texts: Table 1 rows sum to 200, Table 3 rows sum to 100 or 200, and Table 4 rows sum similarly. This is consistent with metrics being computed on the same labeled examples that drove acquisition, training, and calibration, i.e., in-sample evaluation. If that is the case, accuracy and AUC-ROC reflect the model's ability to fit or calibrate to labels it has already seen, not its ability to generalize to new texts. Since the abstract and conclusion make a generalization claim ('as few as 200 manually labeled texts'), the existence of a proper held-out test set is the load-bearing support for that claim, and it is missing from the manuscript as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Conformal Active Learning framework for extracting labels from unstructured medical (or proxy) texts. The pipeline trains a classifier on an initial labeled set, uses label-conditional conformal scores to rank unlabeled texts by uncertainty, clusters the top uncertain candidates for diversity, obtains manual labels for selected points, and retrains. Experiments on Amazon product reviews with four labels compare TF-IDF+XGBoost and DeBERTaV3 under budgets of 100 or 200 labels and claim strong performance, a privacy/lightweight advantage, and a conformal coverage guarantee. The paper also releases code and describes the OLIM web interface.","tokens_in":8711,"tokens_out":6152,"duration_ms":55860,"significance":"If the evaluation were sound, the framework would be a useful practical contribution: it is model-agnostic, lightweight, open-source, and addresses privacy by allowing on-site deployment. The authors are honest about the proxy-dataset limitation and about the weaker deep-learning results. The conformal theorem is standard and the included proof is correct under exchangeability. However, as reported, the evaluation does not establish the central generalization claim because no held-out test set is described, and the conformal guarantee is not valid for the adaptive acquisition loop. The rare-label results are unstable and the strongest numbers rely on tuned initialization and splits. The contribution is therefore not yet substantiated.","major_comments":[{"comment":"The evaluation never defines a held-out test set. The only split described is the statement that \"20% of the labeled data was randomly assigned to the validation dataset in each cycle to calibrate the conformal prediction component.\" The Yes/No columns in Tables 1, 3, and 4 sum exactly to the number of manually labeled texts (100 or 200), which is consistent with metrics being computed on the same labeled examples that drove training and calibration. Because the abstract and conclusion claim generalization from 200 labels, the results as presented do not support that claim; a proper test split disjoint from the active-learning loop is required.","section":"Experimental Setup, Parameters"},{"comment":"Theorem 1 is a standard label-conditional conformal guarantee and its proof in Appendix A is correct under the stated exchangeability assumption. However, the framework applies it inside an active learning loop: the calibration set is a random 20% subset of the adaptively labeled data, and the \"new\" points are selected from the unlabeled pool using the same model's conformal scores (Eq. (4) and the section \"Clustering Selection for Manual Labeling\"). These points are not exchangeable with the calibration data, so the asserted coverage guarantee P[Y_new in C_alpha(X_new) | Y_new = y] >= 1 - alpha does not follow from Theorem 1. The manuscript should either modify the algorithm to preserve exchangeability or state clearly that the guarantee applies only to a single induction step under an idealized static distribution.","section":"Conformal Scores / Theorem 1"},{"comment":"The central claim in the abstract that the framework achieves strong performance with as few as 200 manually labeled texts is not supported by the rare-label results. In Table 3, for Damaged product with 200 manual labels the AUC-ROC is 0.50 ± 0.50, and it reaches 0.88 only after starting with 40 keyword-selected pre-labeled texts, which is not the claimed \"as few as 200 manually labeled texts\" in an unsupervised-start sense. With 100 labels, Pet product has AUC-ROC 0.79 ± 0.07 (or 0.95 only with a tuned 50/50 high/low uncertainty split), and Table 2 shows DeBERTaV3 AUC-ROC values of 0.44 and 0.42. The claim of strong general performance is therefore overstated and needs to be scoped to favorable conditions.","section":"Results, Table 3"},{"comment":"The random-selection baselines in Table 4 (rows marked with observation 7) are also evaluated without a described held-out test set, and their reported standard deviations are very wide (AUC-ROC 0.90 ± 0.10 for Pet and 0.50 ± 0.50 for Damaged). As written, the comparison between the active learning framework and random selection is therefore not a controlled comparison of generalization performance; it is an in-sample comparison with high variance. Reporting confidence intervals over repeated runs on a fixed held-out test set would be needed to make the claimed advantage credible.","section":"Experimental Setup / Results, random baseline"}],"minor_comments":[{"comment":"There is a grammatical error: \"Experiments show that our framework achieving strong performance\" should read \"Experiments show that our framework achieves strong performance.\"","section":"Abstract"},{"comment":"The caption says \"active leaning cycle\" and should say \"active learning cycle.\"","section":"Figure 1 caption"},{"comment":"The text says \"we compute a thresholds for each label\" and should say \"we compute a threshold for each label.\"","section":"Conformal Scores"},{"comment":"\"ktop splitted 30/70\" and similar phrases should be \"ktop split 30/70.\"","section":"Experimental Setup, Parameters"},{"comment":"The statement that MIMIC-III \"does not contain electronic health record (EHR) data\" is inaccurate; MIMIC-III contains free-text clinical notes, which are EHR data. This factual error should be corrected or the intended distinction (structured vs. unstructured EHR data) should be stated precisely.","section":"Experimental Setup, Dataset"},{"comment":"Appendix C refers to \"Figure 1,\" \"Figure 2,\" and \"Figure 3\" for the OLIM interface, which conflicts with the numbering of Figure 1 in the main text and will confuse readers.","section":"Appendix C"}],"recommendation":"reject","confidential_remarks":"I concur with the reader's assessment. The manuscript is not ready for publication in its current form: the missing held-out evaluation and the invalid adaptive conformal guarantee are load-bearing issues, not presentation issues. A resubmission with a proper test split, corrected conformal claims, and repeated-run details could be considered, but the current evidence does not support the abstract's central claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: this paper builds a reasonable pipeline — label-conditional conformal scores drive uncertainty ranking, k-means adds diversity, and the loop retrains on selected labels — and the code is open source. But the central claim, that 200 labels are enough for strong classification, is not backed by the experiments as written. The manuscript never describes a held-out test set disjoint from the actively selected labels. The Yes/No counts in Tables 1, 3, and 4 sum exactly to the number of manually labeled texts, which strongly suggests the metrics are computed in-sample. If that's true, the accuracy and AUC-ROC reflect fitting to labels already seen, not generalization. That's a load-bearing flaw, not a nit.\n\nWhat's genuinely new: the specific combination of conformal-score uncertainty sampling with clustering diversity is not in the cited literature, as far as I can tell, and the model-agnostic, on-premise framing is sensible. The appendix proof of the conformal theorem is standard and correctly stated, though it assumes exchangeability, which the active learning loop violates. The authors do not address that gap. There's also a factual error: MIMIC-III absolutely contains EHR data (it's a critical care database), so the proxy-dataset justification contains a false premise.\n\nWhere this is salvageable: with a proper independent test split, an honest discussion of the conformal guarantee under adaptive sampling, and corrected claims about MIMIC-III, the framework could be a useful engineering contribution. The rare-label results are unstable (AUC 0.50 to 0.88 across configurations), which is acknowledged but not fully explained.\n\nWho this is for: someone working on active learning for clinical text mining might get ideas from the pipeline and the open-source tooling, but they should not rely on the reported numbers. I'd send it to peer review only with the expectation of heavy revision; as it stands, the evidence does not support the abstract's promise.\n\nRecommendation: don't desk-reject outright — a serious referee should get a look — but the referee should be told to check whether any held-out evaluation exists. If the analysis stands as written, REJECT. If the authors redo the evaluation properly, it could be accepted after major changes.","headline":"Sensible engineering combination of conformal prediction and active learning, but the central claim of strong performance from 200 labels is unsupported by the evaluation as written.","tokens_in":9220,"tokens_out":2193,"would_cite":false,"duration_ms":20289,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that combining active learning with label-conditional conformal prediction lets a lightweight classifier match deep learning on unstructured text using only 200 manually labeled examples.","keywords":["active learning","conformal prediction","label-conditional conformal prediction","electronic health records","epidemiological surveillance","text classification","uncertainty quantification","low-annotation learning"],"falsifier":"Take the released code and reproduce Table 1's 200-label protocol while holding out a random 20% of the corpus as a test set that is never used for selection, training, or calibration; if the held-out AUC-ROC falls significantly below the reported values (for example, below 0.7 for the Pet product label), the paper's central claim of strong generalization from 200 labels is refuted.","tokens_in":8225,"feed_emoji":"🩺","tokens_out":8941,"duration_ms":71029,"temperature":0.7,"pith_summary":"The paper tries to establish that a loop of uncertainty-ranked, cluster-diversified manual labeling can train a text classifier to strong performance with very few annotations, and that the same loop provides a per-label statistical coverage guarantee. The proposed Conformal Active Learning framework first trains a model on a small seed of guessed or pre-labeled texts, then repeatedly scores unlabeled texts by conformity, selects a diverse batch for specialist review, and retrains. The reported results on an Amazon-reviews proxy for electronic health records reach accuracy of 0.77–0.92 and AUC-ROC of 0.75–0.94 for four labels using only 200 manual labels with a TF-IDF + XGBClassifier, while a DeBERTaV3 transformer is not stronger. If this holds, institutions with commodity hardware could run privacy-preserving epidemiological surveillance on their own text without sending patient data externally.","feed_headline":"200 labels reach strong classification of free text","feed_subtitle":"Conformal active learning lets on-site lightweight models match deep learning with a fraction of the labels.","key_machinery":"The engine is a label-conditional conformal score $s(x,y)=1-\\hat{p}(y|x)$, where $\\hat{p}(y|x)$ is the model's predicted probability that text $x$ has label $y$. For each label $y$, a validation set provides a threshold $t_{y,\\alpha}$ equal to the $(1-\\alpha)$ empirical quantile of scores of correctly labeled points, and a new text's prediction set is $C_\\alpha(x)=\\{y : s(x,y)\\le t_{y,\\alpha}\\}$. The framework ranks unlabeled texts by the mean of these scores over labels in the prediction set, keeps the top $k_{\\mathrm{top}}$, clusters them with $k$-means on the model's embeddings, and sends the point closest to each of $k_{\\mathrm{cluster}}$ centroids for manual labeling. This selection loop is what couples uncertainty quantification with targeted annotation, and a fraction of low-uncertainty texts can be mixed in to validate easy cases.","core_discovery":"The paper's central claim is that a conformal active learning framework can extract structured labels from unstructured text with as few as 200 manually labeled examples. In the reported experiments, a TF-IDF + XGBClassifier achieves accuracy between 0.77 and 0.92 and AUC-ROC between 0.75 and 0.94 across four labels (Pet product, Drinkable product, Low quality product, Damaged product) after 200 specialist annotations, and this performance is competitive with a DeBERTaV3 transformer. The key to the result is the selection loop: rank unlabeled texts by mean conformal score, cluster the most uncertain ones by embedding, and label the cluster centroids, while mixing in a fraction of low-uncertainty texts. A pre-labeled seed (e.g., 20 or 40 texts, possibly found by keyword search) is especially important for rare labels.","pith_inferences":["If the mechanism is causal, the conformal score ranking should be replaced or augmented by a measure of expected information gain, which could yield even larger label savings; this is a testable variant the paper does not explore.","The dataset being product reviews rather than medical text means the claimed privacy and clinical utility rest on an analogy; validating the same loop on a properly de-identified EHR corpus with a temporal split would be the decisive test.","The clustering step uses k-means on model embeddings, so the diversity of selected points depends on the embedding model's quality; one could test whether using a domain-specific embedding (e.g., from a clinical language model) changes the labeling efficiency."],"forward_implications":["Health institutions could deploy the framework on commodity hardware and produce structured symptom labels from free-text records with roughly 200 specialist annotations per label.","The conformal prediction layer gives each text a set of acceptable labels at a user-chosen confidence level, so outputs can be used to flag uncertain cases for human review instead of committing to a single label.","Including a 30/70 or 50/50 mix of low- and high-uncertainty texts in the labeling queue, plus a small pre-labeled seed, materially raises accuracy and AUC-ROC, especially for rare or subjective labels.","A simple TF-IDF + XGBClassifier pipeline can perform competitively with a transformer model, which argues for prioritizing deployment simplicity and data privacy over model size."],"supporting_citations":[{"why":"Supplies the label-conditional conformal prediction guarantee that underpins Theorem 1.","marker":"V ovk 2012"},{"why":"Provides the quantile lemma used to complete the proof of the coverage guarantee in the appendix.","marker":"Tibshirani et al. 2019"},{"why":"The active learning survey that motivates the iterative selection loop.","marker":"Settles 2009"},{"why":"The Amazon product reviews dataset used as the electronic-health-record proxy in all experiments.","marker":"Rumi 2021"},{"why":"The XGBClassifier that delivers the reported strong results with a lightweight model.","marker":"Chen and Guestrin 2016"},{"why":"DeBERTaV3, the deep learning baseline that the lightweight model is compared against.","marker":"He et al. 2020"}],"fun_headline_variants":["200 labels match deep learning for EHR text mining","Conformal active learning: 200 labels rival transformers","Simple models, 200 labels, compete with deep learning","On-site privacy: 200 labels for strong text classification","200 labeled texts make lightweight models viable for EHRs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim assumes that the reported accuracy and AUC-ROC values were measured on a held-out test set that the active learning loop never touched; the paper describes a validation split for calibration but no separate test split, so if the numbers came from the training or calibration texts, the strong-generalization claim would collapse.","fun_headline_variants_meta":{"raw":{"variants":["200 labels match deep learning for EHR text mining","Conformal active learning: 200 labels rival transformers","Simple models, 200 labels, compete with deep learning","On-site privacy: 200 labels for strong text classification","200 labeled texts make lightweight models viable for EHRs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000337,"raw_usage":{"total_tokens":1837,"prompt_tokens":889,"completion_tokens":948,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":871}},"tokens_in":505,"tokens_out":948,"duration_ms":8665,"temperature":1.0,"reasoning_tokens":871,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T05:47:44.193228+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the released code and reproduce Table 1's 200-label protocol while holding out a random 20% of the corpus as a test set that is never used for selection, training, or calibration; if the held-out AUC-ROC falls significantly below the reported values (for example, below 0.7 for the Pet product label), the paper's central claim of strong generalization from 200 labels is refuted.","supporting_citations":[{"cited_title":"J.; Barber, R","cited_arxiv_id":null,"evidence_quote":"Provides the quantile lemma used to complete the proof of the coverage guarantee in the appendix."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The active learning survey that motivates the iterative selection loop."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Amazon product reviews dataset used as the electronic-health-record proxy in all experiments."}],"review_version":1}