{"id":"15629cd4-f1e8-46fd-ab33-255a773236f8","arxiv_id":"2508.19077","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new SOAP-based physician intent taxonomy and labeled dialogue dataset, with benchmarks showing models classify intents accurately but fail to predict SOAP-category transitions.","lead":"This paper builds a new annotated dataset of over 5,200 doctor-patient dialogue turns, labeling each physician utterance with a 20-class intent taxonomy based on the SOAP clinical framework. It reports that AI models classify medical intents well but predict transitions between clinical reasoning phases poorly, and that intent-based filtering can improve medical summarization.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Turn-level train/test splits are not grouped by dialogue; with only 207 Aci-bench dialogues, same-dialogue context almost certainly leaks into training and inflates next-intent prediction and transition-failure results.","rationale":"I read the paper as making three intertwined claims: (1) a reusable SOAP-aligned intent dataset, (2) a benchmark showing models classify intents well but predict next intents poorly, and (3) a downstream summarization boost from intent filtering. The dataset itself is a real contribution, and the paper is transparent about limitations (role-played dialogues, no decoder fine-tuning). The most load-bearing assumption for the benchmark and transition-failure analysis is that the splits measure generalization across dialogues, not just across turns. The manuscript never states that splits are grouped by dialogue, and the sample counts in Tables 2–3 are turn counts, so the default reading is a turn-level stratified split. Given only 207 source dialogues, turn-level splitting almost guarantees dialogue overlap between train and test. This is not a theoretical nit: the next-intent model sees up to five preceding turns, and the training set can contain the same dialogue's later turns/context, so the model can exploit dialogue-level identity. The summarization filter is also exposed to the same dialogues in training and evaluation, compounding the problem. I do not see another concern as load-bearing: the 'first work' novelty claim is a literature-coverage claim that cannot be settled from inside the paper, and the annotation accuracy 81.13% is reported but with limited inter-annotator detail. The split issue, however, directly threatens the numerical evidence for the central 'models fail at transitions' conclusion. The reader's weakest assumption already identified this, so we agree; the appropriate outcome is unchanged: accept conditionally, with the dialogue-grouped re-run as the condition.","tokens_in":14559,"tokens_out":5588,"duration_ms":59550,"concrete_test":"Check the released code/data: compute the set of dialogue IDs in the next-intent training and test splits. If any dialogue appears in both, rerun the GatorTronS next-intent fine-tuning and evaluation using a grouped split (e.g., GroupShuffleSplit on dialogue_id) with identical hyperparameters, and compare macro-AP/AUROC and the Section 5.3 transition-error pattern. A material drop (e.g., AP falling below ~0.30) or disappearance of the 'late/premature transition' errors would confirm that the reported numbers are leakage-inflated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The main empirical finding—GatorTronS reaches 0.69 macro-AP on intent classification but only 0.37 on next-intent prediction, and models 'fail to identify transitions between SOAP categories' (Abstract)—depends on the split described in Section 4. The paper says only 'we apply stratified sampling' and reports turn-level sample counts (5292/3886/646/760). It does not state that all turns from one Aci-bench dialogue are kept in the same split. Aci-bench contains just 207 dialogues with roughly 27 turns each; random turn assignment makes it very likely that most test dialogues also appear in training. For the next-intent task, the input is up to five preceding doctor-patient turns; if a nearby segment of the same dialogue was seen in training, the model can copy the trajectory pattern instead of predicting from general physician behavior. This would inflate AUROC/AP and change the transition-error analysis in Section 5.3. The same overlap contaminates the summarization filter, which is trained on the same dialogues that are then filtered and summarized. The taxonomy and released annotations remain useful, but the central trajectory benchmarks need to be re-run with dialogue-level grouping.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a physician-intent annotation dataset built on the Aci-bench doctor-patient dialogues. With input from medical professionals, the authors define a 20-class taxonomy organized under SOAP categories, annotate or verify more than 5,000 turns via Prolific, and release the data, guidelines, and code. They benchmark encoder and decoder-only models on two tasks: classifying the intent of a single physician utterance, and predicting the intent of the next physician turn given up to five preceding turns. They also analyze common SOAP trajectories, evaluate how well a fine-tuned model reconstructs dialogue sequences, and test whether intent-based filtering improves medical-dialogue summarization. The main reported findings are that fine-tuned clinical encoders (especially GatorTronS) perform well on intent classification but poorly on next-intent prediction, that models fail to detect SOAP phase transitions, and that filtering improves summarization for several settings.","tokens_in":14889,"tokens_out":3881,"duration_ms":46601,"significance":"If the claims hold, the main contribution is a reusable, SOAP-aligned physician-intent corpus with annotation guidelines, which would be of clear value to clinical dialogue research. The public release of the dataset is a concrete strength, as is the effort to involve medical professionals in taxonomy design and verification. The trajectory analysis is also potentially useful for designing diagnostic dialogue systems. However, the evaluation claims rest on experiment-design choices that are currently not fully controlled; in particular, the lack of dialogue-level separation between training and test splits and the overlap between the summarization filter's training data and the summarized dialogues could materially change the reported numbers. The central dataset contribution is not invalidated, but the benchmarking and downstream-filtering conclusions require re-validation with cleaner protocols.","major_comments":[{"comment":"The train/validation/test splits are described only as 'stratified sampling' over the 5,292 labeled turns, with no statement that all turns from one Aci-bench dialogue are kept in the same split. Aci-bench contains only 207 dialogues with roughly 27 turns each, so turn-level random splitting almost certainly places turns from the same dialogue in both training and test. For next-intent prediction, whose input is up to five preceding doctor-patient turns, the model can then memorize a specific dialogue's trajectory from nearby training turns, inflating the reported GatorTronS 0.85 AUROC / 0.37 AP and undermining the Section 5.3 transition-error analysis. Please re-run the benchmarks using dialogue-level grouped splits and report the number of dialogues in each split, or explicitly justify why turn-level splits are valid.","section":"Section 4.1, Tables 2-3"},{"comment":"The intent filter is the GatorTronS model fine-tuned on the 5,292 labeled turns from the same Aci-bench dialogues that are later filtered and summarized. This creates a circular evaluation: the filter has already seen these dialogues during training, so the reported 'significant boost' from filtering may be optimistic and may not transfer to unseen dialogues. The summarization experiments should be repeated with a filter trained on a dialogue-disjoint set, or at minimum the paper should report results for filtering with a model trained only on held-out dialogues. This is essential to support the abstract's claim that intent filtering improves summarization.","section":"Section 6"},{"comment":"The data verification is summarized by a single accuracy figure of 81.13%, with the remaining 19.87% reviewed and 'unresolved disagreements' removed. It is not stated how many annotators labeled each turn, what inter-annotator agreement was, how accuracy was computed (before or after removing unresolved disagreements), or how many samples were removed and according to what criteria. Because the dataset is the paper's central contribution, this level of detail is insufficient. Please report full annotation statistics, including per-class precision/recall against the in-house labels, Cohen's kappa or similar, and the exact exclusion rule.","section":"Section 3.1"},{"comment":"The conclusion that 'the model does not learn phase transitions' is supported only by qualitative inspection of four reconstructed sequences in Figure 5. The procedure for generating the 'Model Output' sequences from the next-intent-prediction model is not fully specified, and no automatic metric quantifies transition errors over the full test set. Since this analysis is a central piece of the paper's main claim that models fail to identify transitions between SOAP categories, please define the reconstruction algorithm and report quantitative transition-error statistics (e.g., precision/recall for SOAP-category changes, timing errors) on the dialogue-level test split.","section":"Section 5.3"}],"minor_comments":[{"comment":"Typo: '1,77' should be '1.77'.","section":"Table 1"},{"comment":"The token-level sequence plots are nearly unreadable at printed size; consider enlarging fonts, using banded SOAP category colors, or separating into panels.","section":"Figure 5"},{"comment":"The summarization setup does not state whether the Aci-bench train/test split from Yim et al. is used; please clarify the split and how the filter's training dialogues relate to the summarization test dialogues.","section":"Section 6"},{"comment":"The phrase 'significant boost' is not backed by statistical significance testing. Report confidence intervals or paired tests for the filter comparisons, especially where gains are small (e.g., GPT-4o assessment-and-plan: 0.50 to 0.51).","section":"Section 6"},{"comment":"The percentage differences, e.g., 'AP drops by 114.89%' for hierarchical fine-tuning, are confusing because the drop is from 0.37 to 0.10, which is a percentage-point reduction of 0.27; please report percentage-point differences or absolute changes.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The dataset and taxonomy are genuinely useful, and the authors should be encouraged to revise. The main risk is that the empirical benchmark results may change substantially once dialogue-level data separation is imposed; this is a correctness issue, not a scope issue. The annotation-verification reporting also needs to be more transparent. I would not reject because the resource contribution stands independently, but the paper's advertised findings about next-intent prediction and summarization filtering should not be accepted in their current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The resource contribution is real. A fine-grained SOAP-based physician intent taxonomy for dialogues, applied to 5,292 turns from Aci-bench, with guidelines, code, and data released—this is the first reusable intent corpus of its kind and will be useful for clinical dialogue work. The characterization findings (doctors spend most turns on Subjective, speak most tokens in Plan, chitchat is omnipresent) are clear and plausible. The intent classification results, with fine-tuned encoders beating zero- and few-shot LLMs, are credible.\n\nThe soft spot is the next-intent prediction benchmark, and it is load-bearing. The paper says only \"stratified sampling\" over 5,292 turns, with no statement that turns from the same Aci-bench dialogue are kept together. With just 207 dialogues, random turn assignment means most test dialogues also appear in training. The next-intent input is up to five preceding turns, so the model can copy trajectory patterns from the same dialogue seen during training. That would inflate the reported AUROC/AP and undermine the transition-error analysis in Section 5.3. The GatorTronS 0.37 AP figure and the claim that models \"fail to identify transitions\" need to be re-run with dialogue-level grouping before they can be trusted. This is fixable, but it is a required fix.\n\nThe summarization filter has a related circularity: the GatorTronS filter is trained on the same Aci-bench dialogues that are later filtered and evaluated. The \"significant boost\" claim is also overstated—gains are mixed and biggest on one task, and BART quality drops on full-note and subjective summarization. The verification process is under-reported: 81.13% accuracy with unresolved disagreements removed is not the same as inter-annotator agreement, and the paper should give IAA and details on the Prolific crowd \"experts.\" The limitations section does honestly note the role-played nature of the source dialogues and the unfine-tuned decoders, which is a point in the authors' favor.\n\nBottom line: the dataset and taxonomy deserve to see the light, and the intent classification results are probably fine, but the trajectory conclusions cannot be evaluated until the split issue is addressed. The paper needs serious peer review with major revision, not a desk reject. I'd send it to review and ask for dialogue-level splits and IAA reporting.","headline":"A genuinely useful SOAP-aligned intent dataset with a solid taxonomy, but the trajectory benchmark numbers are suspect because the train/test split is done at the turn level, so same-dialogue context almost certainly leaks.","tokens_in":15302,"tokens_out":1015,"would_cite":true,"duration_ms":12744,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims to be the first to study physician intent trajectories in doctor-patient dialogues, releasing a 5,292-turn SOAP-labeled corpus and showing that current models classify single intents well but predict next intents poorly.","keywords":["physician intent trajectories","SOAP taxonomy","doctor-patient dialogue","medical intent classification","next intent prediction","dialogue summarization","clinical NLP dataset"],"falsifier":"Re-split the 5,292 turns by dialogue so all turns of one Aci-bench conversation stay in the same fold, fine-tune GatorTronS on next-intent prediction, and compare macro-AP with the reported 0.37. A large drop would show the transition numbers were inflated by within-dialogue leakage; a small drop would confirm the transition-failure result.","tokens_in":14544,"feed_emoji":"🩺","tokens_out":11032,"duration_ms":96472,"temperature":0.7,"pith_summary":"This paper tries to establish that doctor-patient dialogues have a measurable structure -- the sequence of physician intents -- and that this structure can be captured with a taxonomy built on the SOAP note framework. The authors annotate more than 5,000 turns from 207 role-played clinical conversations with 20 fine-grained physician intents, verify the labels with medical experts, and release the corpus with guidelines. Benchmarking shows that a fine-tuned clinical encoder classifies a single doctor utterance's intent well (0.69 macro average precision) but predicts the next intent far worse (0.37), indicating that current models understand the content of medical dialogue better than its trajectory. The paper also reports common trajectory patterns and shows that filtering dialogues by predicted intent before summarization improves generated notes for most sections. If correct, this provides a reusable resource and a specific diagnosis of where medical dialogue systems need improvement.","feed_headline":"First corpus maps physician intent trajectories in clinic talks","feed_subtitle":"5,000+ labeled turns show models know doctor intents but miss SOAP phase transitions; filtering boosts summaries.","key_machinery":"The machinery is the SOAP intent trajectory: each doctor utterance is labeled with one or more of 20 intents nested under the clinical note categories Subjective, Objective, Assessment, and Plan. This taxonomy converts unstructured dialogue into a discrete state sequence, making trajectory analysis and prediction possible. The next-intent task feeds up to five preceding doctor-patient turns into the same fine-tuned encoders, and a hierarchical fine-tuning variant first predicts the SOAP category and then masks intents outside that category; comparing the two tasks is what exposes the transition bottleneck.","core_discovery":"The central discovery is that a clinical conversation can be read as a trajectory through SOAP phases, and that current models learn the phases but not the transitions. The paper builds a 20-intent taxonomy with clinicians, annotates 5,292 doctor-patient turns from the Aci-bench role-played dialogues, and verifies labels with crowd-sourced medical experts (81.13% agreement). Fine-tuned GatorTronS reaches macro-AP 0.69 on intent classification but only 0.37 on next-intent prediction; reconstruction experiments show it misses SOAP phase boundaries, predicting transitions late or early and defaulting to linear paths on non-linear dialogues. The paper concludes models understand dialogue structu","pith_inferences":["The corpus is naturally suited to a phase-boundary detection task: predict the turn where the doctor switches SOAP categories. The paper's error analysis suggests this is where models fail, so a dedicated transition model could improve downstream trajectory reconstruction.","Because Lab Examination is often confused with Radiology and Physical Examination, adding a test-type entity-recognition auxiliary loss would likely sharpen both intent classification and the summarization filter.","A testable extension: rank consultations by the model's transition uncertainty and filter only the uncertain ones before summarization; the paper's verbosity results predict that non-linear and chitchat-heavy dialogues would gain most.","Applying the same annotation protocol to non-role-played or de-identified real clinical dialogues would test how much the trajectory patterns and filtering gains transfer beyond synthetic consultations."],"forward_implications":["A public 5,292-turn, 20-intent corpus with annotation guidelines gives clinical NLP a shared test bed, so future physician-intent models can be compared on the same trajectories.","Because intent classification (0.69 macro-AP) clearly outperforms next-intent prediction (0.37 macro-AP), practical decision support should use intent classifiers for structuring and coding conversations, not for anticipating the doctor's next move.","Common trajectory patterns -- Subjective-dominant openings, brief Assessment phases, repeated Plan negotiation loops, chitchat scattered throughout -- give differential-diagnosis systems a prior to plan against.","Intent filtering before summarization improves average decoder-only summarization by 5.39 percent, with the largest gain in objective-exam sections, so intent-aware preprocessing is a cheap, effective step for clinical note generation.","The model's linear-default behavior on non-linear dialogues means evaluations of medical dialogue systems should include follow-up and annual-exam conversations, not only first-visit complaints."],"supporting_citations":[{"why":"Supplies the Aci-bench corpus of 207 role-played doctor-patient dialogues that all annotation, trajectory, and summarization experiments are built on.","marker":"[38]"},{"why":"Defines the SOAP note framework that the 20-intent taxonomy adapts to dialogue turns.","marker":"[32]"},{"why":"GatorTronS is the clinical encoder fine-tuned for both tasks and yields the paper's best intent-classification and transition scores.","marker":"[37]"},{"why":"BiomedBERT is the second clinical encoder baseline, used for comparison on both tasks.","marker":"[14]"},{"why":"Llama-3.1-8B-Instruct is the primary decoder-only baseline evaluated zero-shot and few-shot.","marker":"[13]"},{"why":"Qwen2.5-7B-Instruct is the second decoder-only baseline, showing that decoder-only models do not reach encoder performance.","marker":"[36]"},{"why":"BART-large is the summarization model fine-tuned and then tested with and without intent filtering.","marker":"[17]"},{"why":"Guided decoding lets decoder-only models emit classification labels in a fixed output space, making their scores directly comparable.","marker":"[33]"}],"fun_headline_variants":["Doctors' intent paths mapped in 5,000+ clinic dialogues","Models read doctor intent but miss SOAP shift points","New dataset tracks physician intent across clinical chats","AI misses phase changes in doctor-patient talks, study finds"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the role-played Aci-bench dialogues stand in for real doctor-patient consultations, a limitation the paper states in its final section; the next-intent numbers additionally assume that the stratified train/test splits do not carry turns from the same dialogue into both sides.","fun_headline_variants_meta":{"raw":{"variants":["Doctors' intent paths mapped in 5,000+ clinic dialogues","Models read doctor intent but miss SOAP shift points","New dataset tracks physician intent across clinical chats","AI misses phase changes in doctor-patient talks, study finds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00015,"raw_usage":{"total_tokens":1064,"prompt_tokens":803,"completion_tokens":261,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":195}},"tokens_in":547,"tokens_out":261,"duration_ms":3358,"temperature":1.0,"reasoning_tokens":195,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:56:17.493050+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-split the 5,292 turns by dialogue so all turns of one Aci-bench conversation stay in the same fold, fine-tune GatorTronS on next-intent prediction, and compare macro-AP with the reported 0.37. A large drop would show the transition numbers were inflated by within-dialogue leakage; a small drop would confirm the transition-failure result.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the SOAP note framework that the 20-intent taxonomy adapts to dialogue turns."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GatorTronS is the clinical encoder fine-tuned for both tasks and yields the paper's best intent-classification and transition scores."},{"cited_title":"Grattafiori, A","cited_arxiv_id":null,"evidence_quote":"Llama-3.1-8B-Instruct is the primary decoder-only baseline evaluated zero-shot and few-shot."},{"cited_title":"Lewis, Y","cited_arxiv_id":null,"evidence_quote":"BART-large is the summarization model fine-tuned and then tested with and without intent filtering."}],"review_version":1}