Pith. sign in

REVIEW 4 major objections 6 minor 16 references

Mining Unstructured Medical Texts With Conformal Active Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that combining active learning with label-conditional conformal prediction lets a lightweight classifier match deep learning on unstructured text using only 200 manually labeled examples.

desk verdict Sensible engineering combination of conformal prediction and active learning, but the central claim of strong performance from 200 labels is unsupported by the evaluation as written. read the letter →

arxiv 2502.04372 v1 pith:CXGLDZWQ submitted 2025-02-05 cs.CL cs.LGstat.ML

Juliano Genari , Guilherme Tegoni Goedert This is my paper · ORCID
classification cs.CLcs.LGstat.ML
keywords activelearningconformalpredictionlabel-conditionalelectronichealthrecordsepidemiologicalsurveillancetextclassificationuncertaintyquantificationlow-annotation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a loop of uncertainty-ranked, cluster-diversified manual labeling can train a text classifier to strong performance with very few annotations, and that the same loop provides a per-label statistical coverage guarantee. The proposed Conformal Active Learning framework first trains a model on a small seed of guessed or pre-labeled texts, then repeatedly scores unlabeled texts by conformity, selects a diverse batch for specialist review, and retrains. The reported results on an Amazon-reviews proxy for electronic health records reach accuracy of 0.77–0.92 and AUC-ROC of 0.75–0.94 for four labels using only 200 manual labels with a TF-IDF + XGBClassifier, while a DeBERTaV3 transformer is not stronger. If this holds, institutions with commodity hardware could run privacy-preserving epidemiological surveillance on their own text without sending patient data externally.

What carries the argument

The engine is a label-conditional conformal score $s(x,y)=1-\hat{p}(y|x)$, where $\hat{p}(y|x)$ is the model's predicted probability that text $x$ has label $y$. For each label $y$, a validation set provides a threshold $t_{y,\alpha}$ equal to the $(1-\alpha)$ empirical quantile of scores of correctly labeled points, and a new text's prediction set is $C_\alpha(x)=\{y : s(x,y)\le t_{y,\alpha}\}$. The framework ranks unlabeled texts by the mean of these scores over labels in the prediction set, keeps the top $k_{\mathrm{top}}$, clusters them with $k$-means on the model's embeddings, and sends the point closest to each of $k_{\mathrm{cluster}}$ centroids for manual labeling. This selection loop is what couples uncertainty quantification with targeted annotation, and a fraction of low-uncertainty texts can be mixed in to validate easy cases.

What would settle it

Take the released code and reproduce Table 1's 200-label protocol while holding out a random 20% of the corpus as a test set that is never used for selection, training, or calibration; if the held-out AUC-ROC falls significantly below the reported values (for example, below 0.7 for the Pet product label), the paper's central claim of strong generalization from 200 labels is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that a conformal active learning framework can extract structured labels from unstructured text with as few as 200 manually labeled examples. In the reported experiments, a TF-IDF + XGBClassifier achieves accuracy between 0.77 and 0.92 and AUC-ROC between 0.75 and 0.94 across four labels (Pet product, Drinkable product, Low quality product, Damaged product) after 200 specialist annotations, and this performance is competitive with a DeBERTaV3 transformer. The key to the result is the selection loop: rank unlabeled texts by mean conformal score, cluster the most uncertain ones by embedding, and label the cluster centroids, while mixing in a fraction of low-uncertainty texts. A pre-labeled seed (e.g., 20 or 40 texts, possibly found by keyword search) is especially important for rare labels.

Load-bearing premise

The central claim assumes that the reported accuracy and AUC-ROC values were measured on a held-out test set that the active learning loop never touched; the paper describes a validation split for calibration but no separate test split, so if the numbers came from the training or calibration texts, the strong-generalization claim would collapse.

Editorial extensions

If this is right

  • Health institutions could deploy the framework on commodity hardware and produce structured symptom labels from free-text records with roughly 200 specialist annotations per label.
  • The conformal prediction layer gives each text a set of acceptable labels at a user-chosen confidence level, so outputs can be used to flag uncertain cases for human review instead of committing to a single label.
  • Including a 30/70 or 50/50 mix of low- and high-uncertainty texts in the labeling queue, plus a small pre-labeled seed, materially raises accuracy and AUC-ROC, especially for rare or subjective labels.
  • A simple TF-IDF + XGBClassifier pipeline can perform competitively with a transformer model, which argues for prioritizing deployment simplicity and data privacy over model size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism is causal, the conformal score ranking should be replaced or augmented by a measure of expected information gain, which could yield even larger label savings; this is a testable variant the paper does not explore.
  • The dataset being product reviews rather than medical text means the claimed privacy and clinical utility rest on an analogy; validating the same loop on a properly de-identified EHR corpus with a temporal split would be the decisive test.
  • The clustering step uses k-means on model embeddings, so the diversity of selected points depends on the embedding model's quality; one could test whether using a domain-specific embedding (e.g., from a clinical language model) changes the labeling efficiency.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a Conformal Active Learning framework for extracting labels from unstructured medical (or proxy) texts. The pipeline trains a classifier on an initial labeled set, uses label-conditional conformal scores to rank unlabeled texts by uncertainty, clusters the top uncertain candidates for diversity, obtains manual labels for selected points, and retrains. Experiments on Amazon product reviews with four labels compare TF-IDF+XGBoost and DeBERTaV3 under budgets of 100 or 200 labels and claim strong performance, a privacy/lightweight advantage, and a conformal coverage guarantee. The paper also releases code and describes the OLIM web interface.

Significance. If the evaluation were sound, the framework would be a useful practical contribution: it is model-agnostic, lightweight, open-source, and addresses privacy by allowing on-site deployment. The authors are honest about the proxy-dataset limitation and about the weaker deep-learning results. The conformal theorem is standard and the included proof is correct under exchangeability. However, as reported, the evaluation does not establish the central generalization claim because no held-out test set is described, and the conformal guarantee is not valid for the adaptive acquisition loop. The rare-label results are unstable and the strongest numbers rely on tuned initialization and splits. The contribution is therefore not yet substantiated.

major comments (4)
  1. [Experimental Setup, Parameters] The evaluation never defines a held-out test set. The only split described is the statement that "20% of the labeled data was randomly assigned to the validation dataset in each cycle to calibrate the conformal prediction component." The Yes/No columns in Tables 1, 3, and 4 sum exactly to the number of manually labeled texts (100 or 200), which is consistent with metrics being computed on the same labeled examples that drove training and calibration. Because the abstract and conclusion claim generalization from 200 labels, the results as presented do not support that claim; a proper test split disjoint from the active-learning loop is required.
  2. [Conformal Scores / Theorem 1] Theorem 1 is a standard label-conditional conformal guarantee and its proof in Appendix A is correct under the stated exchangeability assumption. However, the framework applies it inside an active learning loop: the calibration set is a random 20% subset of the adaptively labeled data, and the "new" points are selected from the unlabeled pool using the same model's conformal scores (Eq. (4) and the section "Clustering Selection for Manual Labeling"). These points are not exchangeable with the calibration data, so the asserted coverage guarantee P[Y_new in C_alpha(X_new) | Y_new = y] >= 1 - alpha does not follow from Theorem 1. The manuscript should either modify the algorithm to preserve exchangeability or state clearly that the guarantee applies only to a single induction step under an idealized static distribution.
  3. [Results, Table 3] The central claim in the abstract that the framework achieves strong performance with as few as 200 manually labeled texts is not supported by the rare-label results. In Table 3, for Damaged product with 200 manual labels the AUC-ROC is 0.50 ± 0.50, and it reaches 0.88 only after starting with 40 keyword-selected pre-labeled texts, which is not the claimed "as few as 200 manually labeled texts" in an unsupervised-start sense. With 100 labels, Pet product has AUC-ROC 0.79 ± 0.07 (or 0.95 only with a tuned 50/50 high/low uncertainty split), and Table 2 shows DeBERTaV3 AUC-ROC values of 0.44 and 0.42. The claim of strong general performance is therefore overstated and needs to be scoped to favorable conditions.
  4. [Experimental Setup / Results, random baseline] The random-selection baselines in Table 4 (rows marked with observation 7) are also evaluated without a described held-out test set, and their reported standard deviations are very wide (AUC-ROC 0.90 ± 0.10 for Pet and 0.50 ± 0.50 for Damaged). As written, the comparison between the active learning framework and random selection is therefore not a controlled comparison of generalization performance; it is an in-sample comparison with high variance. Reporting confidence intervals over repeated runs on a fixed held-out test set would be needed to make the claimed advantage credible.
minor comments (6)
  1. [Abstract] There is a grammatical error: "Experiments show that our framework achieving strong performance" should read "Experiments show that our framework achieves strong performance."
  2. [Figure 1 caption] The caption says "active leaning cycle" and should say "active learning cycle."
  3. [Conformal Scores] The text says "we compute a thresholds for each label" and should say "we compute a threshold for each label."
  4. [Experimental Setup, Parameters] "ktop splitted 30/70" and similar phrases should be "ktop split 30/70."
  5. [Experimental Setup, Dataset] The statement that MIMIC-III "does not contain electronic health record (EHR) data" is inaccurate; MIMIC-III contains free-text clinical notes, which are EHR data. This factual error should be corrected or the intended distinction (structured vs. unstructured EHR data) should be stated precisely.
  6. [Appendix C] Appendix C refers to "Figure 1," "Figure 2," and "Figure 3" for the OLIM interface, which conflicts with the numbering of Figure 1 in the main text and will confuse readers.

Circularity Check

1 steps flagged · score 6.0 of 10

Reported accuracy and AUC-ROC appear to be computed on the manually labeled texts themselves, so the headline generalization claim rests on in-sample numbers.

  1. fitted input called prediction [Abstract; Experimental Setup (Parameters); Results, Tables 1, 3, and 4]
    ""All experiments were conducted with 100 or 200 manually labeled texts... Additionally, 20% of the labeled data was randomly assigned to the validation dataset in each cycle to calibrate the conformal prediction component and ensure reliable uncertainty estimation." Table 1 caption: "Final performance with XGBClassifier for the proposed labels after 200 manual labels using our framework" (Yes/No sums 62/138, 70/130, 46/154, 39/161, each equaling 200)."

    The only split the paper describes is the 20% validation subset used for conformal calibration; no independent held-out test set is defined anywhere in the paper. In Tables 1, 3, and 4 the Yes/No columns sum to the number of manually labeled texts in every row (200 for Table 1; 100 or 200 in Tables 3 and 4), and the same pattern holds for the random-selection baseline. The Abstract's load-bearing claim, "strong performance with as few as 200 manually labeled texts," therefore rests on accuracy and AUC-ROC computed within the same labeled pool used for training and threshold calibration.

full rationale

The conformal mathematics is not circular: Theorem 1 is a standard external result whose proof invokes Lemma 1 of Tibshirani et al. (2019), and the thresholds t_{y,alpha} are obtained by ordinary quantile calibration rather than tuned to manufacture coverage. There are no self-citations in the paper, so the self-citation and imported-uniqueness patterns do not apply. The framework is an honest assembly of known components (active learning, label-conditional conformal prediction, k-means diversity selection), and the paper is transparent about its proxy dataset ("there are no publicly available open medical databases for this purpose") and about failures on rare labels (AUC-ROC 0.5 for Damaged with 100 labels). The circular step is in the evaluation of the central claim: the Parameters section states that all experiments use 100 or 200 manually labeled texts and that the only split is "20% of the labeled data... assigned to the validation dataset in each cycle to calibrate the conformal prediction component"; no held-out test set is ever described. The Yes/No columns of Tables 1, 3, and 4 sum to exactly the number of manually labeled texts in every row, indicating that the reported accuracy and AUC-ROC are computed within the actively labeled pool (training/calibration, or at most its biased 20% subset), not on independent texts. The Abstract's claim of "strong performance with as few as 200 manually labeled texts" is therefore supported only by in-sample fit, which by construction cannot establish generalization to unseen EHR text; this is the reduction that drives the score. A secondary validity concern, weighed but not scored as a separate circular step, is that Theorem 1's exchangeability premise is asserted for the active-learning loop even though the labeled texts are selected precisely because of their conformal scores, so the claimed 90% coverage guarantee is not established on the loop as run. Because the missing independent evaluation set is the load-bearing support for the central claim, rather than a peripheral citation issue, the score is 6 rather than 0-2; a properly described held-out test would dissolve this circularity.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The framework relies on a standard conformal calibration theorem, several hand-chosen hyperparameters that drive the reported results, and an unverified evaluation premise. No new entities are introduced. The main load-bearing axioms are exchangeability of calibration data, transferability of Amazon review results to EHR text, and the existence of an independent test set, none of which is established in the paper.

free parameters (6)
  • ktop = 500
    Number of high-uncertainty points considered each cycle; hand-chosen and affects which points can be labeled.
  • kcluster = 6
    Number of manually labeled points per cycle; directly sets the labeling budget.
  • high/low uncertainty split = 30/70 (also 50/50, 70/30 tested)
    Fraction of selected points from high vs low uncertainty; the best-performing split varies by label and the reported headline results use the favorable 30/70 split.
  • initial pre-labeled texts = 20 or 40 with keyword search positives
    The rare-label results improve dramatically when seeded with 40 keyword-selected positives, so the '200 labels' claim is conditional on this hand-built seed.
  • confidence level alpha = 0.10
    Confidence level for conformal sets; affects threshold values and which labels enter each prediction set, hence the uncertainty ranking.
  • validation split fraction = 20%
    Fraction of labeled data held out for conformal calibration each cycle; no independent test split is described.
assumptions (6)
  • domain assumption The calibration and deployment data are exchangeable (Theorem 1).
    Used to assert P[Y_new in C_alpha(X_new) | Y_new=y] >= 1-alpha; active learning selection and retraining violate exchangeability.
  • domain assumption Amazon product reviews are a valid proxy for unstructured EHR text.
    The framework is evaluated only on this proxy; no medical text is tested, so the central deployment claim rests on transferability.
  • domain assumption Reported metrics are computed on a held-out test set.
    The paper never states such a split; Yes/No columns sum to the number of labels, so in-sample evaluation is plausible.
  • domain assumption Model embeddings form meaningful clusters for diversity sampling.
    k-means selection assumes the embedding space separates text types so that centroid-near points are diverse and informative.
  • standard math Lemma 1 of Tibshirani et al. 2019 gives the label-conditional quantile guarantee.
    The appendix proof of Theorem 1 relies on this external lemma without deriving it.
  • ad hoc to paper The uncertainty score S_X in Eq. (4) ranks informativeness for active learning.
    No theorem connects mean conformity score over the prediction set to labeling value; it is a heuristic selection rule.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mining Unstructured Medical Texts With Conformal Active Learning." pith.science (2026). https://pith.science/paper/CXGLDZWQ

@misc{pith2026250204372,
  author       = {Pith},
  title        = {Pith review of: Mining Unstructured Medical Texts With Conformal Active Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CXGLDZWQ}},
  note         = {Machine review of arXiv:2502.04372}
}
read the original abstract

The extraction of relevant data from Electronic Health Records (EHRs) is crucial to identifying symptoms and automating epidemiological surveillance processes. By harnessing the vast amount of unstructured text in EHRs, we can detect patterns that indicate the onset of disease outbreaks, enabling faster, more targeted public health responses. Our proposed framework provides a flexible and efficient solution for mining data from unstructured texts, significantly reducing the need for extensive manual labeling by specialists. Experiments show that our framework achieving strong performance with as few as 200 manually labeled texts, even for complex classification problems. Additionally, our approach can function with simple lightweight models, achieving competitive and occasionally even better results compared to more resource-intensive deep learning models. This capability not only accelerates processing times but also preserves patient privacy, as the data can be processed on weaker on-site hardware rather than being transferred to external systems. Our methodology, therefore, offers a practical, scalable, and privacy-conscious approach to real-time epidemiological monitoring, equipping health institutions to respond rapidly and effectively to emerging health threats.

Figures

Figures reproduced from arXiv: 2502.04372 by the authors.

Figure 1
Figure 1. Diagram of the active leaning cycle. privacy-preserving deployment, and iterative machine learn￾ing for healthcare. Conformal Active Learning Our main goal is to infer labels Y (for example, if a patient has or not a symptom) for unstructured texts X (EHRs, etc.). To achieve this goal we propose a Conformal Active Learn￾ing framework, see [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Convergence of AUC-ROC for the Pet product label, using the XGBBoost model, with only high uncertainty and a 70/30 mix of high and low uncertainty. texts (20), we managed to achieve strong results with just 200 manual labels; see [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 1
Figure 1. Label management interface with label management, progress, CSV download and upload and [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figures from the paper (2 more)
Figure 2
Figure 2. Figure 2: Search interface with term inclusion/exclusion syntax. [PITH_FULL_IMAGE:figures/full_fig_p009_2.png]
Figure 3
Figure 3. Figure 3: Interaction page for specialists. Texts are chosen to be labeled based on configurable priorities. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 10 canonical work pages

  1. [1]

    N.; and Bates, S

    Angelopoulos, A. N.; and Bates, S. 2021. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511

    arXiv 2021

  2. [2]

    Chen, T.; and Guestrin, C. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, 785--794

    2016

  3. [3]

    Ding, T.; Angelopoulos, A.; Bates, S.; Jordan, M.; and Tibshirani, R. J. 2024. Class-conditional conformal prediction with many classes. Advances in Neural Information Processing Systems, 36

    work page 2024

  4. [4]

    He, P.; Liu, X.; Gao, J.; and Chen, W. 2020. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654

    arXiv 2020

  5. [5]

    Huang, J.; Xi, H.; Zhang, L.; Yao, H.; Qiu, Y.; and Wei, H. 2024. Conformal Prediction for Deep Classifier via Label Ranking. In Salakhutdinov, R.; Kolter, Z.; Heller, K.; Weller, A.; Oliver, N.; Scarlett, J.; and Berkenkamp, F., eds., Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Resea...

    work page 2024

  6. [6]

    Ibrahim, N. K. 2020. Epidemiologic surveillance for controlling Covid-19 pandemic: types, challenges and implications. Journal of infection and public health, 13(11): 1630--1638

    work page 2020

  7. [7]

    E.; Pollard, T

    Johnson, A. E.; Pollard, T. J.; Shen, L.; Lehman, L.-w. H.; Feng, M.; Ghassemi, M.; Moody, B.; Szolovits, P.; Anthony Celi, L.; and Mark, R. G. 2016. MIMIC-III, a freely accessible critical care database. Scientific data, 3(1): 1--9

    2016

  8. [8]

    D.; Nkengasong, J.; Kettler, H.; Tongio, I.; Gay-Andrieu, F.; Escadafal, C.; Piot, P.; Peeling, R

    Kelly-Cirino, C. D.; Nkengasong, J.; Kettler, H.; Tongio, I.; Gay-Andrieu, F.; Escadafal, C.; Piot, P.; Peeling, R. W.; Gadde, R.; and Boehme, C. 2019. Importance of diagnostics in epidemic and pandemic preparedness. BMJ global health, 4(Suppl 2): e001179

    work page 2019

Show all 16 references
  1. [9]

    Romano, Y.; Sesia, M.; and Cand\` e s, E. J. 2020. Classification with valid and adaptive coverage. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20. Red Hook, NY, USA: Curran Associates Inc. ISBN 9781713829546

    2020

  2. [10]

    Rumi, A. 2021. Amazon Product Reviews. https://www.kaggle.com/datasets/arhamrumi/amazon-product-reviews. Accessed: 2024-11-10

    2021

  3. [11]

    Settles, B. 2009. Active learning literature survey. CS Technical Reports

    2009

  4. [12]

    B.; Parrish, R

    Thacker, S. B.; Parrish, R. G.; and Trowbridge, F. L. 1988. A method for evaluating systems of epidemiological surveillance. World Health Statistics Quarterly 1988; 41 (1): 11-18

    1988

  5. [13]

    J.; Barber, R

    Tibshirani, R. J.; Barber, R. F.; Cand \`e s, E. J.; and Ramdas, A. 2019. Conformal Prediction Under Covariate Shift. In Neural Information Processing Systems

    2019

  6. [14]

    Vovk, V. 2012. Conditional validity of inductive conformal predictors. In Asian conference on machine learning, 475--490. PMLR

    2012

  7. [15]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  8. [16]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.