{"id":"8669b147-6a54-4e1e-a739-389f763eadcc","arxiv_id":"2503.05701","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A BERT model distilled from GPT-4 labels sorts patient portal messages into Admin and Clinical categories with 88.85% agreement with the labeler, and was deployed through Epic's Nebula cloud platform.","lead":"Researchers trained an AI to sort patient portal messages into administrative or clinical categories, using GPT-4 to generate training labels and a smaller BERT model to run the classification in production. If the accuracy holds in practice, the tool could lighten the message load that contributes to physician burnout, and it was already deployed inside Johns Hopkins' Epic system.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 88.85% accuracy is measured against GPT-4-generated test labels; without physician-validated test labels, the headline accuracy may reflect agreement with the teacher, not true classification accuracy.","rationale":"The reader's weakest assumption is exactly the load-bearing concern I identify: the test-set labels are GPT-4-generated and were not physician-validated. This is the single most important threat to the central claim because every headline metric (accuracy, sensitivity, specificity, F1) is computed against those labels. The paper does provide one piece of independent support—three physicians evaluated 2,000 messages used for prompt selection and GPT-4 achieved 0.99 agreement—so the approach is not baseless. However, that validation does not cover the test set, and it is not evidence that GPT-4 labels are error-free on the broader distribution. The verdict CONDITIONAL is appropriate: the paper should be accepted only if the authors supply human-validated test-set metrics, or the claims should be explicitly downgraded to 'agreement with GPT-4.' I also note secondary unresolved inconsistencies (training/validation/test sizes sum to 40,702 while the text says ~36k; '92%' vs 88.85%; deployment outcome data absent), but these do not change the primary concern. The reader's proposed conditions for upgrade are reasonable and match my analysis.","tokens_in":8455,"tokens_out":3216,"duration_ms":81017,"concrete_test":"Have two or three physicians independently label a stratified random sample of 300-500 messages from the 3,454-message test set, blinded to both GPT-4 and BERT labels. Compute BERT accuracy, sensitivity, specificity, and F1 against the physician majority label, and compute a 95% confidence interval. Also compute GPT-4's agreement with physicians on the same sample to detect systematic label bias. If the human-validated accuracy is within 5 points of 88.85% and the confidence interval includes the reported value, the headline claim survives; if it is materially lower, the reported accuracy should be reinterpreted as teacher-agreement, not clinical accuracy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim is that the distilled BERT model achieves 88.85% accuracy on the test set. The test set labels were generated by GPT-4-32K ('A prompt-based, fine-tuned version of GPT4-32K was used to label the dataset for model distillation'). The only human validation reported is on the 2,000-message prompt-selection set, where three physicians evaluated GPT-4 predictions (Table 4, E2: 0.99 accuracy). The paper never states that physicians validated the 3,454 test-set labels. Therefore, the reported sensitivity, specificity, and F1 are agreement rates between BERT and GPT-4, not accuracy against a human ground truth. If GPT-4 has systematic labeling errors—especially class bias on ambiguous 'Admin' vs 'Clinical' messages—BERT will inherit those errors, and true performance against physician judgment is unknown. The 2,000-message physician check is real independent evidence that GPT-4 aligns with physicians on a selection set, but that set was used for prompt selection, was evenly balanced, and does not establish the test-set label quality. Without a human-labeled test sample, the load-bearing assumption that '88.85% accuracy' means accuracy on the actual clinical task is unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript describes OPTIC, a pipeline that uses GPT-4 few-shot prompting to label roughly 36K patient portal messages drawn from 405,487 MyChart encounters at Johns Hopkins Medicine, then distills those labels into a BERT classifier that separates \"Admin\" from \"Clinical\" messages. The BERT model is reported to achieve 88.85% accuracy, 88.29% sensitivity, 89.38% specificity, and F1 0.8842 on a 3,454-message test set. The paper also presents BERTopic topic modeling of the test messages and describes deployment through Epic's Nebula Cloud Platform. The central claimed contribution is a scalable, deployed triaging tool for reducing physician InBasket burden.","tokens_in":8599,"tokens_out":6015,"duration_ms":49641,"significance":"If the reported performance holds against physician-labeled ground truth, the work would be a useful practical contribution: the dataset is substantially larger than those used in prior portal-message classification studies, and the integration with Epic's Nebula Cloud Platform is an unusual demonstration of real-world deployment. The physician review of the GPT-4 prompt-selection set (Table 4, E2: 0.99 accuracy) is a genuine strength, as is the explicit reporting of sensitivity, specificity, and F1. However, the headline evaluation is against GPT-4-generated labels, not human ground truth, and the deployment claims lack outcome metrics; both need to be addressed before the practical-effectiveness conclusion is justified.","major_comments":[{"comment":"The reported headline metrics (accuracy 88.85%, sensitivity 88.29%, specificity 89.38%, F1 0.8842) are computed against a test set whose labels were generated by the same GPT-4 pipeline used to create the training labels: 'A prompt-based, fine-tuned version of GPT4-32K was used to label the dataset for model distillation.' These numbers are therefore agreement rates between BERT and GPT-4, not classification accuracy against an independent ground truth. The only physician validation reported (Table 4, E2) was performed on a separate 2,000-message set used for prompt selection, and the manuscript does not state that the 3,454 test labels were physician-reviewed. Because systematic GPT-4 labeling errors on ambiguous messages would be inherited by BERT and then scored as correct, the central '88.85% accuracy' claim is not yet supported. Please provide a physician-labeled test sample with reported agreement statistics, or explicitly reframe the claims as agreement with GPT-4 labels and temper the corresponding effectiveness statements.","section":"Results, first paragraph; Abstract"},{"comment":"The 'Possible Administrative' and 'Possible Clinical' groups are defined by metadata (EMP/SER flags and Order/Note linkage) rather than by message content. Because the GPT-4 labeling and BERT distillation training pools are sampled from these groups, any systematic disagreement between the metadata-based categories and the true content-based categories will be encoded in the model. Furthermore, the 292,291 'Uncategorized' messages are excluded from training and from the reported evaluation, yet a deployed triaging system will encounter exactly such messages. The paper should report the GPT-4 label distribution within each retrospective group, quantify agreement between metadata categories and GPT-4 labels on a physician-reviewed sample, and evaluate the BERT model on a sample from the Uncategorized group; without these, the generalization claim to the full patient-message population is unsupported.","section":"Retrospective Group Analysis and Cohort Selection"},{"comment":"The claim that deployment through Epic's Nebula Cloud Platform 'demonstrat[es] its practical effectiveness in healthcare settings' is not supported by any deployment outcome data in the manuscript. No metrics are reported on message volume processed, provider time saved, physician satisfaction, or comparison with the pre-deployment workflow. The deployment is a valuable feasibility statement, but the effectiveness claim requires either quantitative deployment evaluation or a more modest formulation.","section":"Abstract; Conclusion"}],"minor_comments":[{"comment":"The text says 'For the validation dataset, we used the same 2,000 messages which were used for GPT prompt validation,' but the Results split reports a 3,387-message validation set; additionally, '~36k messages' is inconsistent with 33,861 + 3,387 + 3,454 = 40,702. Please clarify the dataset composition and the role of the 2,000 prompt-selection messages in the BERT validation.","section":"Model distillation; Results"},{"comment":"The claim that the model achieves over 80% accuracy for 58 of 81 topics lacks information on topic sizes or confidence intervals; please include the number of messages per topic and the accuracy distribution.","section":"Results, BERTopic analysis"},{"comment":"The manuscript does not report the BERT model variant, fine-tuning hyperparameters, or checkpoint selection procedure; for reproducibility, please provide these details and a data/code availability statement.","section":"GPT Prompt Analysis and Model Distillation"},{"comment":"The clustering figures for Administrative and Clinical groups are referenced without axis labels or a description of the embedding projection, making them difficult to interpret.","section":"Figure 4"},{"comment":"Reference [23] appears to have an incomplete or incorrect author list; please verify the entry against the original publication.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a realistic application-oriented manuscript. The editor should ensure that the authors treat the GPT-4 label circularity seriously: without a physician-labeled test sample, the headline number is better described as teacher-student agreement. The deployment description is a distinctive feature but should not be used as a substitute for outcome evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: OPTIC is a deployment story, not a methodological advance. The genuinely new artifact is a BERT classifier, distilled from GPT-4 labels, running in Epic's Nebula cloud for patient portal message triage at Johns Hopkins. That is real, and it is the paper's main value.\n\nWhat it does well: the dataset is large (405K primary care encounters), the retrospective weak-labeling via Epic metadata (EMP/SER flags, order/note linkage) is a sensible way to get scale, and the prompt-selection step is honestly reported. Three physicians evaluated 2,000 messages with the chosen prompt (E2) and hit 0.99 agreement with GPT-4. That is genuine evidence that the teacher knows the task on that sample. Distilling down to BERT for CPU inference is the right engineering move.\n\nThe soft spots, in order. The reported 88.85% accuracy is measured against test labels generated by the same GPT-4 pipeline that produced the training labels. So it is an agreement rate with the teacher, not accuracy against physician ground truth. The 2,000-message physician check was for prompt selection; it does not cover the 3,454 test messages. If GPT-4 has systematic biases, BERT inherits them and true accuracy is unknown. That is the load-bearing weakness, and it is fixable: label a held-out test set by physicians.\n\nSecond, no baseline is reported. Prior classifiers on portal messages are cited, but never run for comparison. A simple rule-based or keyword baseline would put 89% in context. Third, there are minor numeric inconsistencies: the distillation section claims 92% accuracy, the Results section reports 88.85%; the split sizes (33,861 + 3,387 + 3,454) do not match the \"~36k\" description. The BERTopic analysis is peripheral and also relies on GPT-4 labels.\n\nWho gets value from this? Applied clinical informatics teams that want a worked template for LLM-to-small-model distillation inside a major EHR. It is not a rigorous evaluation paper, but it deserves a serious referee: the deployment is tangible, the prompt validation is a good-faith effort, and the central flaw is addressable. My recommendation: send to peer review, require a physician-labeled test set and at least one baseline before acceptance.","headline":"A real deployment story with a flawed headline metric: the reported 88.85% accuracy is BERT agreeing with GPT-4 labels, not with physician ground truth.","tokens_in":9320,"tokens_out":2849,"would_cite":false,"duration_ms":25402,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A GPT-4-distilled BERT model sorts patient messages with about 89% accuracy.","keywords":["patient portal messages","InBasket triage","GPT-4 data labeling","model distillation","BERT classifier","administrative versus clinical","clinical operations","EHR deployment"],"falsifier":"Take a random sample of several hundred test-set messages, have multiple physicians independently label each one as administrative or clinical, and compare BERT's predictions, GPT-4's labels, and the physicians' majority vote. If BERT's agreement with the human labels is substantially below the reported 88.85%, or if GPT-4 and the physicians disagree on a large share of messages, then the reported accuracy measures agreement with an unvalidated teacher rather than real-world performance.","tokens_in":8146,"feed_emoji":"📨","tokens_out":6420,"duration_ms":54353,"temperature":0.7,"pith_summary":"The paper claims that a small, deployable language model can take over a large share of patient-portal triage: a BERT classifier distilled from GPT-4 labels separates administrative from clinical patient messages with 88.85% accuracy on a 3,454-message test set. The motivation is concrete: physicians receive dozens to hundreds of inbox messages a day, and prior studies cited in the paper put the administrative share above 50%, meaning much of that volume could be handled by non-clinical staff. Using roughly 405,000 patient portal encounters, the authors used GPT-4 few-shot prompting with 200 physician-reviewed examples to label about 36,000 messages, then trained BERT on those labels. The resulting model was deployed through the EHR vendor's Nebula cloud platform, and topic-level analysis shows it stays above 80% accuracy on 58 of 81 message topics. If the result holds, the pipeline offers a practical route from a large language model to a cheap, CPU-runnable classifier already embedded in the EHR workflow.","feed_headline":"GPT-4-labeled BERT sorts patient messages with ~89% accuracy","feed_subtitle":"A low-cost BERT model triages admin vs clinical inbox messages, backed by GPT-4 labels and deployed in the EHR.","key_machinery":"The load-bearing mechanism is model distillation from a large language model to a small one. GPT-4-32K is given a few-shot prompt containing 200 messages, evenly split between administrative and clinical examples that were selected by topic-based sampling and reviewed by a physician, and is asked to classify each new message and explain its reasoning. Among four prompt variants, this configuration scored 0.99 on a 2,000-message physician-evaluated validation set, so its labels were used to train a BERT classifier. BERT (Bidirectional Encoder Representations from Transformers) is a pretrained transformer that learns a compact, CPU-runnable representation of text; distilled on roughly 33,000 GPT-4-labeled messages, it reproduces most of the teacher's triage judgment at a fraction of the inference cost. BERTopic, an unsupervised topic model, is then applied to the test set to show that accuracy remains above 80% across most of the 81 identified topics.","core_discovery":"The paper's central claim is that a production-ready triage classifier can be built by using GPT-4 as an annotator and BERT as the distilled student model. On the task of labeling a patient's first message in an encounter as either administrative (scheduling, refills, insurance forms) or clinical (symptoms, test results, treatment questions), the distilled BERT model reaches 88.85% accuracy, 88.29% sensitivity, 89.38% specificity, and an F1 score of 0.8842. The same model is deployed through the EHR vendor's Nebula SaaS platform, meaning the classifier operates inside the same system that hosts patient portal inboxes rather than as a standalone research prototype. The authors position this as an improvement over prior patient-message classifiers, which either used smaller datasets or required higher-cost models, and as a response to the post-pandemic surge in portal messages.","pith_inferences":["The true accuracy against human ground truth is still unknown: the reported 88.85% is agreement with GPT-4, and the paper does not state that physicians validated the test labels. A small human-labeled benchmark would settle this.","The binary admin/clinical split may understate real-world triage difficulty, because many portal messages mix clinical content with administrative requests; a third 'mixed' category or an urgency tier would be a natural extension.","The retrospective metadata groups (employee/service-role flags for administrative, order/note linkage for clinical) are plausible but imperfect proxies; if those groups mislabel a substantial share of messages, the distilled BERT inherits the bias.","The deployment claim would be strengthened by reporting how the model behaves in live use, for example the rate at which providers override its routing, which the paper does not yet provide."],"forward_implications":["A lightweight BERT model can run on CPU inside the EHR, making automated triage affordable at the scale of a large health system that processes over a million inbox messages a year.","Routing administrative messages to non-clinical staff could remove more than half of the inbox volume physicians currently see, since prior studies cited by the paper put the administrative share above 50%.","The same GPT-4-labeling-then-distillation recipe can be reused for other message taxonomies or other specialties, provided new few-shot exemplars are curated and physician-validated.","Deployment through the EHR vendor's Nebula platform means the classifier is available where messages already live, so it can be tested by real outpatient providers rather than only in batch experiments.","Topic-level analysis suggests the model is not tuned to a few frequent message types; it maintains over 80% accuracy on 58 of 81 topics, which supports broader rollout within primary care."],"supporting_citations":[{"why":"Documents the 157% increase in patient messages during the pandemic, establishing the workload problem the system addresses.","marker":"[2]"},{"why":"Reports that physicians receive dozens to 150+ InBasket messages per day, motivating the need for automated triage.","marker":"[3]"},{"why":"Supplies the BERT architecture that the distilled classifier is built on.","marker":"[12]"},{"why":"Provides the earlier rule-based versus machine-learning comparison for classifying patient portal messages that this work extends.","marker":"[13]"},{"why":"Describes a prior health-system patient-message classifier with an AUROC of 0.96, serving as an accuracy baseline for this approach.","marker":"[23]"},{"why":"Supplies BERTopic, the unsupervised topic model used to measure per-topic accuracy across the test set.","marker":"[25]"}],"fun_headline_variants":["GPT-4 labels, BERT triages patient messages with 89% accuracy","Distilled BERT sorts admin vs clinical messages at 89% accuracy","GPT-4 teaches BERT to triage patient inbox messages at 89%","Patient message triage: GPT-4 labels, BERT classifies at 89%","Low-cost BERT triages patient messages using GPT-4 labels, 89% accurate"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that GPT-4's labels on the test messages are correct enough to count as ground truth; only the 2,000-message validation set used for prompt selection was checked by three physicians, and the test-set labels themselves were not human-validated.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4 labels, BERT triages patient messages with 89% accuracy","Distilled BERT sorts admin vs clinical messages at 89% accuracy","GPT-4 teaches BERT to triage patient inbox messages at 89%","Patient message triage: GPT-4 labels, BERT classifies at 89%","Low-cost BERT triages patient messages using GPT-4 labels, 89% accurate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1444,"prompt_tokens":994,"completion_tokens":450,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":342}},"tokens_in":610,"tokens_out":450,"duration_ms":4640,"temperature":1.0,"reasoning_tokens":342,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T10:41:44.792528+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of several hundred test-set messages, have multiple physicians independently label each one as administrative or clinical, and compare BERT's predictions, GPT-4's labels, and the physicians' majority vote. If BERT's agreement with the human labels is substantially below the reported 88.85%, or if GPT-4 and the physicians disagree on a large share of messages, then the reported accuracy measures agreement with an unvalidated teacher rather than real-world performance.","supporting_citations":[{"cited_title":"Assessing the impact of the COVID -19 pandemic on clinician ambulatory electronic health record use","cited_arxiv_id":null,"evidence_quote":"Documents the 157% increase in patient messages during the pandemic, establishing the workload problem the system addresses."},{"cited_title":"How We’re Improving Physicians’ Messaging Experience Through Digital Tools,","cited_arxiv_id":null,"evidence_quote":"Reports that physicians receive dozens to 150+ InBasket messages per day, motivating the need for automated triage."},{"cited_title":"Artificial intelligence to organize patient portal messages: a journey from an ensemble deep learning text classification to rule-based named entity recognition","cited_arxiv_id":null,"evidence_quote":"Describes a prior health-system patient-message classifier with an AUROC of 0.96, serving as an accuracy baseline for this approach."}],"review_version":1}