Pith. sign in

REVIEW 4 major objections 5 minor 42 references

Actively evaluating and learning the distinctions that matter: Vaccine safety signal detection from emergency triage notes

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A human-in-the-loop active-learning workflow trains a vaccine-safety classifier for emergency triage notes that reaches an F1-score of 0.97 on deployment-environment data.

desk verdict A real deployment case study whose headline F1 is computed on a model-selected, partly training-contaminated evaluation set, so the 0.97 claim is not a valid performance estimate. read the letter →

arxiv 2507.18123 v1 pith:IQLG3OLS submitted 2025-07-24 cs.AI cs.CL

Sedigh Khademi , Christopher Palmer , Muhammad Javed , Hazel Clothier , Jim Buttery , Gerardo Luis Dimaguila , Jim Black This is my paper
classification cs.AIcs.CL
keywords activelearningnaturallanguageprocessingvaccinesafetyadverseeventsfollowingimmunizationemergencydepartmenttriagenotescounterfactualdataaugmentationlargemodelssyndromicsurveillance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a rare-event classifier for vaccine-safety signals can be built quickly from emergency-department triage notes without a large pre-existing annotated corpus. The authors start from about 700 records selected by topic diversity, then run four rounds of human-in-the-loop active learning in which uncertainty sampling, counterfactual data augmentation, and evaluation on deployment-environment data steadily improve the model. On a final evaluation set of 1,806 records drawn from the deployment stream, the best model reaches an F1-score (the harmonic mean of precision and recall) of 0.97, up from 0.90 for the best from-scratch round-4 checkpoint and from 0.74 for keyword pattern matching. The significance is practical: if the result holds, post-licensure vaccine surveillance can be stood up in weeks rather than months, using a modest amount of expert labeling time.

What carries the argument

The machinery is an iterative human-in-the-loop active-learning loop rather than a single formula. It begins with a meaning-based clustering of the unlabeled notes, from which a small balanced seed set (about 700 records) is sampled; in each later round, uncertainty sampling—choosing the records the model is least sure about—picks what the expert labels, human label-flipping creates synthetic counterfactual pairs that isolate the distinction that matters (a symptom must be textually linked to a recent vaccination), and model checkpoints are evaluated on unlabeled deployment-environment records rather than only on the validation split. This loop carries the argument: each round's added records and synthetic examples are what turn an initially imprecise model (F1 0.295 at round 1) into a precise one (F1 0.97 at round 4).

What would settle it

Take a fresh random sample of triage notes from the same deployment stream over the same period, have clinicians label those notes without seeing model predictions, and score the final model on that fully held-out set; if its F1 is well below 0.97, or if the random sample reveals AEFI cases the model's own selection missed, the active-evaluation estimate is refuted.

Watch

Extended reading notes

Core claim

The core claim is that combining active learning with human-guided counterfactual augmentation and active evaluation solves the data-scarcity problem for detecting adverse events following immunization in triage text. The authors assemble an initial balanced training set by topic-modelling sentence embeddings over a vaccine-filtered pool of 11,060 notes, label 700 records, and train a biomedical large language model. In each subsequent round, the model's positive predictions and low-confidence negatives on an unlabeled deployment-environment pool are labeled by experts; false positives and uncertain negatives are added to training data, and label-flipped synthetic copies teach the model which textual cues actually connect a symptom to a recent vaccination. By round 4 the final model scores F1 0.97 on the 1,806-record deployment-environment evaluation set (precision 0.955, recall 0.991), compared with 0.90 for the best from-scratch round-4 checkpoint and 0.742 for keyword pattern matching, with gains driven mostly by precision. Over one month of operational use, it flagged 286 records, of which 266 were confirmed AEFI, and a pattern-matching review found no missed positives.

Load-bearing premise

The 1,806-record evaluation set—hand-labeled by the authors from the models' own positive and low-confidence predictions, with some of those records later folded back into training—must actually represent the deployment stream; if it does not, the reported F1 of 0.97 is not a reliable estimate of real-world performance.

Editorial extensions

If this is right

  • A classifier for a rare, clinically important label can be cold-started from roughly 700 expert-labeled notes selected by topic diversity and then improved through iterative uncertainty sampling, without waiting for a large labeled corpus.
  • Counterfactual label-flipping lets the human expert teach the model the distinction that matters—whether the presenting complaint is explicitly linked to a recent vaccination—so precision can rise from 0.72 at round 2 to 0.96 at the final round while recall stays above 0.99.
  • Evaluation on deployment-environment data, rather than on the validation split alone, is what exposes false-positive tendencies and drives the useful training-data additions; validation F1 stayed in a narrow 0.92–0.94 band across rounds while deployment F1 rose from 0.82 to 0.97.
  • The final model can be run operationally: over one month of regional emergency-department records (145,852 notes) it flagged 286 presentations as possible AEFI, of which 266 were confirmed, and a pattern-matching check found no missed positives.
  • Because the model learns the local language of the participating emergency departments, the deployed classifier is expected to be specific to that jurisdiction; what transfers is the active-learning and active-evaluation pipeline, not the model weights.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: because the evaluation set was built from the model's own high-confidence predictions and low-confidence negatives, the reported 0.97 F1 is likely optimistic; a random held-out sample would quantify the gap.
  • Inference: the hand-written label-flipping step could be automated with an LLM generating counterfactual pairs at scale, but synthetic artifacts would need careful auditing before trusting the precision gains.
  • Inference: the same active-learning and active-evaluation loop should transfer to other rare-event triage-screening tasks—such as adverse drug events or self-harm detection—where the bottleneck is the same: few true positives, many look-alike negatives, and no ground-truth test set.
  • Inference: the authors' reactions-based classifier idea, paired with the vaccination-aware model, would allow syndrome-level monitoring without requiring every note to mention a vaccine.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper describes a practical workflow for building a classifier that detects potential adverse events following immunization (AEFI) from emergency department triage notes. The authors use BERTopic-based diversity sampling to create an initial 700-record training set, then iterate through four rounds of uncertainty-based active learning, counterfactual data augmentation, and what they call active evaluation on a deployment-environment text pool. They report that the final model reaches an F1-score of 0.972 on a 1,806-record evaluation set drawn from the deployment pool, compared with 0.883 for the best round-3 checkpoint and 0.900 for a round-4 model trained from scratch, and they claim the classifier is now deployed in practice.

Significance. If the reported performance were valid, this would be a useful contribution to vaccine safety surveillance, showing how a small amount of expert labeling and active learning can produce a deployable classifier on scarce medical text. The paper has real strengths: it uses a large real-world triage corpus, explicitly discusses labeling guidelines and human-in-the-loop decisions, provides an error analysis, and acknowledges the absence of a ground-truth test set. However, the central quantitative claim is not supported by the evaluation design. The 1,806-record evaluation set is constructed from the models' own positive predictions, uncertain negatives, and keyword-pattern matches, and it partially overlaps the training data. As a result, the reported recall and F1 are biased upward, and the headline comparison of 0.972 versus 0.900 is not a valid estimate of comparative classifier performance on unseen data. The methodological narrative remains interesting, but the paper needs a properly sampled independent evaluation or a substantial reinterpretation of its metrics before the central claim can be accepted.

major comments (4)
  1. [§3.5, §4, Table 4] The central performance claim (final F1 0.972 vs. 0.883/0.818 in earlier rounds) is computed on an evaluation set that is not an independent sample of the deployment distribution. The 1,806-record set was assembled by labeling the best checkpoints' positive predictions, their uncertain negatives (<90% probability), and keyword-pattern matches for potential false negatives. This construction excludes high-confidence negatives and, more importantly, any true positives that no model predicted and that pattern matching failed to surface; recall and F1 are therefore computed over a truncated label set and are biased upward. The paper is transparent in §5.2 that no ground truth test set existed, but the Results and Abstract do not carry this caveat. I recommend either labeling a random sample of the deployment pool for evaluation or explicitly downgrading Table 4 to a measure of model agreement rather than classifier performance.
  2. [§3.3, §3.5, Table 3] The evaluation set overlaps the training data in a way that further inflates the reported scores. In each round, false positives and uncertain negatives were added to the training set, and in rounds 3 and 4 newly labeled records from the deployment-environment text pool were added to training (Table 3 shows the training set growing to 1,457 records by round 4). The same or overlapping labeled records then form part of the 1,806-record evaluation set used for Table 4. This direct contamination means the F1 reflects the model's consistency with the labels it helped generate rather than performance on unseen data. An independent holdout set, annotated under the same guidelines without model-driven selection, is needed to support the deployment claim.
  3. [§5.2, error analysis] The one-month error analysis in the deployment environment has the same selection problem: the authors reviewed the 286 records the model predicted as AEFI and pattern-matched keyword negatives, but did not systematically search for true positives that were neither predicted nor matched by keywords. The conclusion that 'we did not find any' false negatives is therefore not supported. This should be reworded from evidence of high recall to a statement about the subset that could be identified, and the possibility of undetected false negatives should be explicitly acknowledged.
  4. [Table 4] With only 107 positive records in the evaluation set, the F1 estimates in Table 4 are not accompanied by confidence intervals; even under an unbiased sampling design, the difference between 0.972 and 0.900 would need interval estimates to be interpretable. Please report confidence intervals (e.g., bootstrap) for all metrics, and clarify how the round-1 row, which is evaluated only on records where its predictions overlapped the round-2 models, should be compared with the other rows.
minor comments (5)
  1. [Abstract] The abstract contains a duplicated phrase: 'active learning, data augmentation, and active learning and evaluation techniques'; please revise for clarity.
  2. [§3.3] The training setup is under-specified: the paper states 9 epochs and batch size 16 but omits learning rate, optimizer, warm-up steps, max sequence length, and early-stopping criteria. Providing these details, or releasing the fine-tuning code, would aid reproducibility.
  3. [Table 4, §3.5] The term 'from scratch' is ambiguous because RoBERTa is initialized from pretrained weights in each round. Please clarify that 'from scratch' means fine-tuning the pretrained base from the round's full training set, not training from random initialization.
  4. [§2] The relation between the AEFI-focused text pool and the deployment-environment text pool is not fully specified; please state whether the 61,600 deployment records overlap the 11,060 filtered records, and how duplicates were handled.
  5. [Table 4, F1Beta] Please define the F1Beta formula and state explicitly that beta > 1 weights recall higher than precision, since the current text says it 'slightly favors recall' but does not give the equation.

Circularity Check

3 steps flagged · score 7.0 of 10

The 0.97 F1 is computed on a model-selected, partially training-contaminated evaluation set, so the central performance claim reduces to the modeling loop itself.

  1. fitted input called prediction [Section 4 (Results), Table 4; also Section 5.2]
    "After eventually labeling all positive predictions and uncertain negatives from the best models from round 2 onwards in the deployment-environment data, we ended up with 1,806 records to use for evaluation – 107 positive and 1699 negative."

    The 1,806-record evaluation set is assembled from the models' own positive predictions and their uncertain negatives (<90% probability), with high-confidence negatives excluded and with true positives that no model predicted and pattern matching missed never labeled. F1 and recall in Table 4 are therefore computed only over records the models already surfaced; they are not estimates over the deployment distribution. The reported F1 of 0.97 is partly a self-consistent outcome of the model's own decision boundary rather than an external benchmark.

  2. fitted input called prediction [Sections 3.3 and 3.5 (Model development and Model evaluation)]
    "In each phase, we took informative examples to add into our training data, being the false positive predictions and the uncertain negatives. ... As training progressed into rounds 3 and 4, we labeled and added examples to the training data from both text pools but relied mostly on the deployment-environment text pool for evaluation."

    The deployment-environment records used for evaluation are the same pool from which labeled examples were added to training in rounds 3 and 4. The final 1,806-record evaluation set therefore overlaps the training data, so the Table 4 scores are contaminated by direct feedback: the model was fine-tuned on labels from the very distribution on which it is then 'evaluated'. This makes the 0.97 F1 a fitted input called a prediction rather than an independent performance estimate.

1 more flagged steps
  1. other [Section 5.2 (Findings)]
    "A crucial factor in our success was tackling the challenge of sample-efficient learning, where no ground truth test set existed and high-quality labeled data for evaluation was limited, by using deployment environment data to perform evaluation of our AL based model."

    The paper explicitly states that no ground-truth test set existed and that the evaluation used the active learner's own selections. This is an admission that the evaluation set is not an independent sample from the deployment environment. Presenting the resulting F1 as a measure of real-world classifier performance is circular with respect to the deployment-readiness claim, because the metric is defined on the model's own output distribution rather than on an externally verifiable ground truth.

full rationale

The central quantitative claim — the further fine-tuned model achieving F1 0.97 versus 0.90 from scratch (§3.5, Table 4) — is evaluated on a set that is (a) constructed from the models' own positive predictions and uncertain negatives, (b) partially fed back into training data in earlier rounds, and (c) explicitly acknowledged to lack a ground-truth test set. Each of these points is directly quotable. High-confidence negatives are excluded from the evaluation set, and true positives that no model predicted and pattern matching missed are never labeled, so recall and F1 cannot be validly estimated. Precision is less biased because all positive predictions were labeled, but the headline F1 is not an external benchmark. The paper is honest about using active evaluation, and the active-learning/data-augmentation methodology itself is not circular; the circularity is confined to the performance claim. Self-citations such as [22] and [23] support design choices but are not the load-bearing reduction here. Because the reported performance reduces by construction to the model's own selection and training feedback, the circularity score is 7.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The central claim depends on several domain assumptions about the data and labeling, and on many hand-chosen hyperparameters. No new entities are introduced. The most consequential assumption is that the evaluation cohort, built from model-selected predictions, is unbiased.

free parameters (7)
  • initial training sample size = 700
    Chosen as a trade-off between label budget and performance; not derived from data.
  • AEFI topic sampling fraction = 60% from AEFI topics, 40% from non-AEFI; 3 records per non-AEFI topic
    Ad hoc to ensure positive examples; section 3.2.
  • uncertainty threshold = 90% probability
    Used to select uncertain negative predictions for labeling; section 3.3.
  • negative:positive ratio cap = 3:2
    Used to balance augmented training data; section 3.3.
  • training epochs = 9
    Default setting; section 3.3.
  • batch size = 16
    Default setting for the GPU; section 3.3.
  • augmented example counts = 100 (round 2), 83 (round 3), 6 removed (round 4)
    Manually chosen based on observed false positives; section 3.4.
assumptions (4)
  • domain assumption Triage notes contain enough information to identify AEFI presentations when vaccination is mentioned.
    The labeling guidelines require an explicit vaccination connection; section 2.2.
  • domain assumption The vaccine-related keyword filter captures all AEFI presentations in the deployed system.
    The text pool is built on keyword matches; the paper acknowledges omissions in section 5.1.
  • domain assumption Labels produced by two authors and a clinician are ground truth.
    No independent reference standard or adjudication against registry data; section 3.2.
  • domain assumption The deployment-environment text pool represents the distribution the model will encounter.
    Used as the primary evaluation data; section 3.5.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Actively evaluating and learning the distinctions that matter: Vaccine safety signal detection from emergency triage notes." pith.science (2026). https://pith.science/paper/IQLG3OLS

@misc{pith2026250718123,
  author       = {Pith},
  title        = {Pith review of: Actively evaluating and learning the distinctions that matter: Vaccine safety signal detection from emergency triage notes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IQLG3OLS}},
  note         = {Machine review of arXiv:2507.18123}
}
read the original abstract

The rapid development of COVID-19 vaccines has showcased the global communitys ability to combat infectious diseases. However, the need for post-licensure surveillance systems has grown due to the limited window for safety data collection in clinical trials and early widespread implementation. This study aims to employ Natural Language Processing techniques and Active Learning to rapidly develop a classifier that detects potential vaccine safety issues from emergency department notes. ED triage notes, containing expert, succinct vital patient information at the point of entry to health systems, can significantly contribute to timely vaccine safety signal surveillance. While keyword-based classification can be effective, it may yield false positives and demand extensive keyword modifications. This is exacerbated by the infrequency of vaccination-related ED presentations and their similarity to other reasons for ED visits. NLP offers a more accurate and efficient alternative, albeit requiring annotated data, which is often scarce in the medical field. Active learning optimizes the annotation process and the quality of annotated data, which can result in faster model implementation and improved model performance. This work combines active learning, data augmentation, and active learning and evaluation techniques to create a classifier that is used to enhance vaccine safety surveillance from ED triage notes.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

42 extracted references · 37 canonical work pages

  1. [1]

    Vaccines 10(5), 722 (2022) Vaccine Safety Signal Detection from Triage Notes 13

    Alami, A., Krewski, D., Mattison, D., Wilson, K., Gravel, C.A., Villeneuve, P.J., Farrell, P.J., Crispo, J.A., Perez-Lloret, S.: Risk of myocarditis and pericarditis among young adults following mrna covid-19 vaccinations. Vaccines 10(5), 722 (2022) Vaccine Safety Signal Detection from Triage Notes 13

    work page 2022

  2. [2]

    ICAART (3) pp

    Andersen, J.S.: Why do we need domain-experts for end-to-end text classification? an overview. ICAART (3) pp. 17–24 (2023)

    work page 2023

  3. [3]

    In: 2021 IEEE 9th International Conference on Healthcare Informatics (ICHI)

    Arnaud, ´E., Elbattah, M., Gignon, M., Dequen, G.: Nlp-based prediction of medical specialties at hospital admission using triage notes. In: 2021 IEEE 9th International Conference on Healthcare Informatics (ICHI). pp. 548–553. IEEE (2021)

    work page 2021

  4. [4]

    Imbalanced Learn- ing: Foundations, Algorithms, and Applications pp

    Attenberg, J., Ertekin, S ¸.: Class imbalance and active learning. Imbalanced Learn- ing: Foundations, Algorithms, and Applications pp. 101–149 (2013)

    work page 2013

  5. [5]

    Australian Government Department of Health and Aged Care: The Australian Immunisation Handbook (2024), https://immunisationhandbook.health.gov.au/, accessed: September 30, 2024

    work page 2024

  6. [6]

    Expert Opinion on Drug Safety 22(8), 659–668 (2023)

    Botsis, T., Kreimeyer, K.: Improving drug safety with adverse event detection using natural language processing. Expert Opinion on Drug Safety 22(8), 659–668 (2023)

    work page 2023

  7. [7]

    In: 2017 IEEE international conference on big data (big data)

    Breck, E., Cai, S., Nielsen, E., Salib, M., Sculley, D.: The ml test score: A rubric for ml production readiness and technical debt reduction. In: 2017 IEEE international conference on big data (big data). pp. 1123–1132. IEEE (2017)

    2017

  8. [8]

    Human Vaccines & Immunotherapeutics 18(6), 2100173 (2022)

    Buttery, J.P., Clothier, H.: Information systems for vaccine safety surveillance. Human Vaccines & Immunotherapeutics 18(6), 2100173 (2022)

    work page 2022

Show all 42 references
  1. [9]

    Vaccine 29(44), 7727–7732 (2011)

    Choe, Y.J., Cho, H., Kim, S.N., Bae, G.R., Lee, J.K.: Serious adverse events follow- ing receipt of trivalent inactivated influenza vaccine in korea, 2003–2010. Vaccine 29(44), 7727–7732 (2011)

    2011

  2. [10]

    Journal of biomedical informatics 46(4), 734–743 (2013)

    Conway, M., Dowling, J.N., Chapman, W.W.: Using chief complaints for syndromic surveillance: a review of chief complaint based classifiers in north america. Journal of biomedical informatics 46(4), 734–743 (2013)

    2013

  3. [11]

    Council for International Organizations of Medical Sciences and Definition, WHO and others: Definition and application of terms for vaccine pharmacovigilance: report of cioms/who working group on vaccine pharmacovigilance (2012)

    2012

  4. [12]

    Expert review of vaccines 13(2), 265–276 (2014)

    Crawford, N.W., Clothier, H., Hodgson, K., Selvaraj, G., Easton, M.L., Buttery, J.P.: Active surveillance for adverse events following immunization. Expert review of vaccines 13(2), 265–276 (2014)

    2014

  5. [13]

    AMIA Summits on Translational Science Proceedings 2020, 152 (2020)

    Dexter, G.P., Grannis, S.J., Dixon, B.E., Kasthurirathne, S.N.: Generalization of machine learning approaches to identify notifiable conditions from a statewide health information exchange. AMIA Summits on Translational Science Proceedings 2020, 152 (2020)

    2020

  6. [14]

    Figueroa, R.L., Zeng-Treitler, Q., Ngo, L.H., Goryachev, S., Wiechmann, E.P.: Active learning for clinical text classification: is it better than random sampling? Journal of the American Medical Informatics Association 19(5), 809–816 (2012)

    2012

  7. [15]

    International Journal of Intelligent Systems 2023(1), 7941878 (2023)

    Fonseca, J., Bacao, F.: Improving active learning performance through the use of data augmentation. International Journal of Intelligent Systems 2023(1), 7941878 (2023)

    2023

  8. [16]

    New England Journal of Medicine 384(22), 2092–2101 (2021)

    Greinacher, A., Thiele, T., Warkentin, T.E., Weisser, K., Kyrle, P.A., Eichinger, S.: Thrombotic thrombocytopenia after chadox1 ncov-19 vaccination. New England Journal of Medicine 384(22), 2092–2101 (2021)

    2021

  9. [17]

    arXiv preprint arXiv:2104.04999 (2021)

    Ha, H., Gupta, S., Rana, S., Venkatesh, S.: Alt-mas: A data-efficient framework for active testing of machine learning algorithms. arXiv preprint arXiv:2104.04999 (2021)

    2021 arXiv

  10. [18]

    In: Proceedings of the 2022 ACM conference on fairness, accountability, and transparency

    Hutchinson, B., Rostamzadeh, N., Greer, C., Heller, K., Prabhakaran, V.: Evalua- tion gaps in machine learning practice. In: Proceedings of the 2022 ACM conference on fairness, accountability, and transparency. pp. 1859–1876 (2022) 14 S. Khademi et al

    2022

  11. [19]

    In: Advances in Knowledge Discovery and Data Mining: 8th Pacific-Asia Conference, PAKDD 2004, Sydney, Australia, May 26-28, 2004

    Kang, J., Ryu, K.R., Kwon, H.C.: Using cluster-based sampling to select initial training set for active learning in text classification. In: Advances in Knowledge Discovery and Data Mining: 8th Pacific-Asia Conference, PAKDD 2004, Sydney, Australia, May 26-28, 2004. Proceeding...

    2004

  12. [20]

    CoRR abs/1909.12434 (2019), http://arxiv.org/abs/1909.12434

    Kaushik, D., Hovy, E., Lipton, Z.C.: Learning the difference that makes a dif- ference with counterfactually-augmenteddata. CoRR abs/1909.12434 (2019), http://arxiv.org/abs/1909.12434

    2019 arXiv

  13. [21]

    In: Proceedings of the 2023 Australasian Computer Sci- ence Week, p

    Khademi, S., Palmer, C., Dimaguila, G.L., Javed, M., Buttery, J., Black, J.: Data augmentation to improve syndromic detection from emergency de- partment notes. In: Proceedings of the 2023 Australasian Computer Sci- ence Week, p. 198–205. ACSW ’23, Association for Computing Ma...

    2023

  14. [22]

    In: Australasian Conference on Data Science and Machine Learning

    Khademi, S., Palmer, C., Javed, M., Dimaguila, G.L., Buttery, J.P., Black, J.: De- tecting asthma presentations from emergency department notes: An active learning approach. In: Australasian Conference on Data Science and Machine Learning. pp. 284–298. Springer (2023)

    2023

  15. [23]

    JMIR AI 3, e54449 (Aug 2024)

    Khademi, S., Palmer, C., Javed, M., Dimaguila, G.L., Clothier, H., Buttery, J., Black, J.: Near real-time syndromic surveillance of emergency department triage texts using natural language processing: Case study in febrile convul- sion detection. JMIR AI 3, e54449 (Aug 2024). ...

    2024 doi

  16. [24]

    JAMA cardiology 6(10), 1196–1201 (2021)

    Kim, H.W., Jenista, E.R., Wendell, D.C., Azevedo, C.F., Campbell, M.J., Darty, S.N., Parker, M.A., Kim, R.J.: Patients with acute myocarditis following mrna covid-19 vaccination. JAMA cardiology 6(10), 1196–1201 (2021)

    2021

  17. [25]

    In: International Conference on Machine Learning

    Kossen, J., Farquhar, S., Gal, Y., Rainforth, T.: Active testing: Sample-efficient model evaluation. In: International Conference on Machine Learning. pp. 5753–

  18. [26]

    In: Proceedings of the Workshop and Tutorial on Interactive Adaptive Learning

    Kottke, D., Calma, A., Huseljic, D., Krempl, G., Sick, B., et al.: Challenges of reliable, realistic and comparable active learning evaluation. In: Proceedings of the Workshop and Tutorial on Interactive Adaptive Learning. pp. 2–14 (2017)

    2017

  19. [27]

    In: 2018 International Joint Conference on Neural Networks (IJCNN)

    Kottke, D., Calma, A., Huseljic, D., Sandrock, C., Kachergis, G., Sick, B.: The other human in the loop–a pilot study to find selection strategies for active learning. In: 2018 International Joint Conference on Neural Networks (IJCNN). pp. 1–8. IEEE (2018)

    2018

  20. [28]

    In: Acm Sigir Forum

    Lewis, D.D.: A sequential algorithm for training text classifiers: Corrigendum and additional data. In: Acm Sigir Forum. vol. 29, pp. 13–19. ACM New York, NY, USA (1995)

    1995

  21. [29]

    In: Proceed- ings of the 3rd clinical natural language processing workshop

    Lewis, P., Ott, M., Du, J., Stoyanov, V.: Pretrained language models for biomedical and clinical tasks: understanding and extending the state-of-the-art. In: Proceed- ings of the 3rd clinical natural language processing workshop. pp. 146–157 (2020)

    2020

  22. [30]

    arXiv preprint arXiv:1301.3781 (2013)

    Mikolov, T.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)

    2013 arXiv

  23. [31]

    Mathematics 12(12), 1898 (2024)

    Moles, L., Andres, A., Echegaray, G., Boto, F.: Exploring data augmentation and active learning benefits in imbalanced datasets. Mathematics 12(12), 1898 (2024)

    2024

  24. [32]

    arXiv preprint arXiv:2011.06874 (2020)

    Mottaghi, A., Sarma, P.K., Amatriain, X., Yeung, S., Kannan, A.: Medical symp- tom recognition from patient text: An active learning approach for long-tailed multilabel distributions. arXiv preprint arXiv:2011.06874 (2020)

    2020 arXiv

  25. [33]

    Olsson, F.: A literature survey of active machine learning in the context of natural language processing. Tech. Rep. 2009:06, RISE, SICS (2009) Vaccine Safety Signal Detection from Triage Notes 15

    2009

  26. [34]

    BMJ open 12(4), e055132 (2022)

    Picard, C.T., Kleib, M., O’Rourke, H.M., Norris, C.M., Douma, M.J.: Emergency nurses’ triage narrative data, their uses and structure: a scoping review protocol. BMJ open 12(4), e055132 (2022)

    2022

  27. [35]

    arXiv preprint arXiv:1908.10084 (2019)

    Reimers, N.: Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 (2019)

    2019 arXiv

  28. [36]

    Journal of the American Medical Informatics Association 29(3), 472–480 (2022)

    Rozova, V., Witt, K., Robinson, J., Li, Y., Verspoor, K.: Detection of self-harm and suicidal ideation in emergency department triage notes. Journal of the American Medical Informatics Association 29(3), 472–480 (2022)

    2022

  29. [37]

    New Eng- land journal of medicine 384(22), 2124–2130 (2021)

    Schultz, N.H., Sørvoll, I.H., Michelsen, A.E., Munthe, L.A., Lund-Johansen, F., Ahlen, M.T., Wiedmann, M., Aamodt, A.H., Skattør, T.H., Tjønnfjord, G.E., et al.: Thrombosis and thrombocytopenia after chadox1 ncov-19 vaccination. New Eng- land journal of medicine 384(22), 2124–...

    2021

  30. [38]

    Morgan & Claypool (2012)

    Settles, B.: Active learning, volume 6 of synthesis lectures on artificial intelligence and machine learning. Morgan & Claypool (2012)

    2012

  31. [39]

    JMIR medical informatics 8(3), e17984 (2020)

    Spasic, I., Nenadic, G., et al.: Clinical text data in machine learning: systematic review. JMIR medical informatics 8(3), e17984 (2020)

    2020

  32. [40]

    Advances in Neural Information Processing Systems 36 (2024)

    Yu, D., Shi, W., Yu, Q.: Actively testing your model while it learns: realizing label- efficient learning in practice. Advances in Neural Information Processing Systems 36 (2024)

    2024

  33. [41]

    arXiv preprint arXiv:2210.10109 (2022)

    Zhang, Z., Strubell, E., Hovy, E.: A survey of active learning for natural language processing. arXiv preprint arXiv:2210.10109 (2022)

    2022 arXiv

  34. [42]

    arXiv preprint arXiv:2108.06332 (2021)

    Zhou, J., Zheng, Y., Tang, J., Li, J., Yang, Z.: Flipda: Effective and robust data augmentation for few-shot learning. arXiv preprint arXiv:2108.06332 (2021)

    2021 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.