{"id":"80b7d2c7-3808-4837-830a-83df69057dd0","arxiv_id":"2506.06737","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A LLaMA3-based triage chatbot trained on GPT-rewritten dialogues, whose claimed superiority over baselines is never directly measured.","lead":"C-PATH is a chatbot that talks with patients about symptoms and suggests a medical department. The paper claims it beats other systems, but the experiments test the training data, not the chatbot itself.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline claim asserts C-PATH's triage performance, but Sections III-E and IV-B evaluate only BERT/ClinicalBERT on static datasets; C-PATH itself never produces an output that is scored, so the central claim has no experimental support.","rationale":"The paper does describe a concrete pipeline and a synthetic-data construction method, and I credit the dataset-construction contribution as potentially useful. However, the central claim is about C-PATH, not about the datasets. Section I states 'Our experiments demonstrate that C-PATH not only produces coherent and accessible dialogues but also offers accurate department-level triage recommendations,' yet Section IV contains no result involving C-PATH's own outputs. The closest quantities are GPTScore ratings of the three dataset variants and BERT/ClinicalBERT classification accuracies on those datasets. A charitable reading might treat dataset classifiability as a proxy for downstream fine-tuning quality, but that inference is never argued, and classifiability of a static corpus does not imply that a separately fine-tuned dialogue model will triage accurately across turns. The GPTScore evaluation also has a secondary circularity issue, since GPT-3.5 generated data_5k_GPT and GPTScore is a GPT-based evaluator, but the more fundamental problem is object-level: the scores are assigned to datasets, not to C-PATH outputs. The paper's own Section V calls for future real-world user studies and RLHF, which corroborates that no direct validation of the deployed system was performed. The concrete test above would directly resolve the concern; if it passed, the paper could be rehabilitated, but as written the central claim is unsupported and the REJECT verdict stands.","tokens_in":12256,"tokens_out":2877,"duration_ms":30142,"concrete_test":"Run C-PATH on a held-out set of patient cases (e.g., 500 DDXPlus cases not used in any training stage) through its actual multi-turn inference loop, record the department it recommends at the end, and compare against ground-truth department labels with clinician adjudication of borderline cases. Include at least two conversational baselines—base LLaMA3-8B with the same prompts and a general-purpose LLM such as GPT-4—and report per-department accuracy, safety flags (e.g., missed red-flag symptoms), and inter-rater agreement. If C-PATH does not beat the base LLaMA3 on held-out triage accuracy, or if no such evaluation can be run because the weights/checkpoints are not released, the superior-performance claim is not established.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim—that C-PATH 'achieves superior performance' and 'outperforms baselines' in conversational triage—rests on evidence that never measures C-PATH. In Section III-E, the authors say the model's primary objective is department recommendation accuracy, but the experiments in Section IV-B report F1/accuracy for BERT and ClinicalBERT on three dataset variants (Table V). Table IV reports GPTScore over the datasets. There is no row for C-PATH, no C-PATH-generated transcript scored by GPTScore, no held-out interaction in which C-PATH recommends a department, and no baseline conversational system. Section IV-C offers one qualitative example, not a measurement. The Abstract and Section V translate dataset-level classifier results into a claim about the fine-tuned model, and the 'domain-specific baselines' comparison is actually BERT vs ClinicalBERT, not C-PATH vs any system. This is a claim-without-derivation gap: even if every reported table is correct, it establishes only that GPT-rewritten DDXPlus conversations are easy for BERT to classify and score highly on GPTScore; it says nothing about whether C-PATH triages accurately or safely.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces C-PATH, a conversational patient assistance and triage system built on LLaMA3-8B, with three claimed contributions: a multi-stage fine-tuning pipeline (medical knowledge injection, conversation tuning, summarization tuning), a GPT-3.5-based data augmentation pipeline that rewrites structured DDXPlus cases into patient-friendly multi-turn conversations, and a dialogue history management mechanism combining sliding-window pruning with optional turn-level summarization. The authors report GPTScore evaluations on the three derived datasets (data_5k_ddxplus, data_5k_artificial, data_5k_GPT) and BERT/ClinicalBERT accuracy on a department-recommendation classification task over those datasets, plus one qualitative example interaction. The abstract and conclusions claim that C-PATH outperforms baselines in conversation quality and triage accuracy.","tokens_in":12459,"tokens_out":4417,"duration_ms":45622,"significance":"If the claims were substantiated, C-PATH would be a relevant contribution to patient-facing LLM triage: the open-source fine-tuning recipe, the use of DDXPlus for synthetic dialogue construction, and the history-management strategy are all useful ingredients, and the paper is explicit about limitations such as the Reversal Curse and department imbalance. The dataset construction pipeline is a potentially reproducible artifact, and the paper reports some concrete hyperparameters and a transparent multi-stage workflow. However, the significance as presented is entirely conditional: the paper never measures C-PATH itself, so the claimed system-level performance has no empirical basis. The paper's strength in dataset creation does not transfer to the conversational triage system it claims to evaluate.","major_comments":[{"comment":"The headline claim that C-PATH 'achieves superior performance' and 'outperforms baselines' is never tested. Section III-E says the model's primary objective is department recommendation accuracy, but the quantitative experiments in Section IV-B (Tables IV and V) evaluate only the three static datasets: GPTScore is applied to the dataset text, and BERT/ClinicalBERT are trained and tested as classifiers on the dataset text. There is no row for C-PATH, no C-PATH-generated transcript scored by GPTScore, no held-out interaction in which C-PATH recommends a department, and no comparison with any conversational baseline. The single example in Figure 8 is an illustration, not a measurement. Consequently, every reported number in the evaluation section is consistent with the hypothesis that C-PATH performs arbitrarily poorly in actual triage.","section":"Abstract, Section IV-B, Section V"},{"comment":"The quality evidence for the GPT-rewritten dataset is circular in a way that is not acknowledged: GPTScore is an LLM-based judge applied to text that was generated by GPT-3.5 Turbo. Because the generator and the judge share stylistic, lexical, and formatting priors, the high understandability and informativeness scores for data_5k_GPT are plausibly inflated relative to a human or independent-metric assessment. The paper reports no human evaluation, no reliability analysis, no confidence intervals, and no repeated scoring runs, so the reported differences across datasets (e.g., SPE falling from 70.02 to 19.92, UND rising from 76.26 to 94.57) should not be treated as statistically meaningful.","section":"Section IV-B, Table IV"},{"comment":"The BERT/ClinicalBERT benchmark does not measure C-PATH's triage capability in any form. It measures whether a standard classifier can predict the department from a static transcript after supervised training. This is a dataset-difficulty measurement, not a system evaluation: C-PATH's multi-turn interaction, follow-up questioning, and recommendation behavior are all absent. The inference drawn from Table V, that 'task-specific fine-tuning can often outperform domain-specific pretraining when applied to conversational healthcare data', is unsupported because the comparison is between two classifiers, not between C-PATH and a domain-specific conversational system. In addition, ClinicalBERT's extremely low F1 on data_5k_ddxplus (0.159) and its inconsistent behavior across datasets suggest a possible mismatch between the pretrained model and the conversational format; this is left unexplained.","section":"Section IV-B, Table V"},{"comment":"The qualitative conversation example is presented as evidence that C-PATH 'engages naturally' and 'allows accurate departmental recommendations', but it does not establish that the transcript was generated by C-PATH, it is not scored, and no error analysis or comparison with other systems is provided. Because this is the only place where the actual system appears in the results, the paper's central claim rests on an anecdote rather than on data.","section":"Section IV-C, Figure 8"}],"minor_comments":[{"comment":"The provenance of data_5k_artificial is stated inconsistently: the text first says the reformulated variants were compiled into data_5k_artificial using GPT-3.5 Turbo, then later calls this set 'manually rewritten questions and answers'. Please clarify whether this dataset is human-written, GPT-generated, or a mix, and describe the human involvement.","section":"Section III-C2"},{"comment":"DDXPlus is described as a dataset of patient cases with symptom hierarchies, not as a set of 'conversations'; the phrase '5000 conversations from the DDXPlus dataset' is inaccurate and should be rephrased as '5000 patient cases from which conversations were constructed'.","section":"Section III-C2"},{"comment":"The hyperparameter table omits LoRA alpha, LoRA dropout, the learning-rate schedule, and warmup steps; including these is necessary for reproducibility of the fine-tuning pipeline.","section":"Table II"},{"comment":"GPTScore is a stochastic metric, but no number of runs or variance measures are reported; the paper should state how many times the scoring was repeated and report standard deviations or confidence intervals.","section":"Section IV-B"},{"comment":"The PAT row values (0.10, 0.10, 0.00) are not interpretable without a statement of the scale and direction (higher = better or lower = better); the current presentation makes a perfect score of 0.00 look like a failure.","section":"Table IV"},{"comment":"References [7] and [8] are the same work (LMFlow); one citation should be removed, and the other should cite the published NAACL 2024 version consistently throughout.","section":"References"},{"comment":"The conclusions repeat the unsupported claim that 'C-PATH outperforms baselines in both conversation quality and triage accuracy'; this sentence should be revised to state only what the experiments actually show, namely dataset-level classifier and GPTScore results.","section":"Section V"}],"recommendation":"reject","confidential_remarks":"The paper is best understood as a dataset-construction and fine-tuning recipe, not as an evaluated conversational system. The evaluation section measures BERT/ClinicalBERT on static datasets, and the GPTScore results are self-referential because the data was generated by the same family of models used for scoring. A revision could possibly be reframed around the dataset pipeline with a human study of the generated conversations, but that would be a substantially different paper. As submitted, the central claim about C-PATH's triage performance is unsupported, which I believe warrants rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: this paper has a real dataset-construction idea but never actually tests its own system. The reader's take is right. C-PATH is a fine-tuned LLaMA3 with LoRA, but every quantitative experiment in Section IV scores the static datasets (GPTScore) or trains BERT/ClinicalBERT for department classification. There's no row for C-PATH, no generated transcript scored, no comparison against any dialogue system. The abstract's claim that C-PATH 'significantly outperforms baselines' is unsupported. The 'domain-specific baselines' are BERT vs ClinicalBERT on the datasets, not C-PATH vs anyone.\n\nWhat's genuinely new: the GPT-3.5 rewriting pipeline that turns structured DDXPlus cases into lay-friendly multi-turn dialogues, with multiple paraphrases and natural patient responses. That could be a useful synthetic dialogue resource. The history management (sliding window + optional summarization + speaker tagging) is standard but sensible. The three-stage fine-tuning is a reasonable recipe. I'd give credit for the data pipeline idea being interesting.\n\nSoft spots, in proportion: the biggest is the missing evaluation of the proposed system. Section III-E says the model's primary objective is department recommendation accuracy, then Table V reports BERT/ClinicalBERT F1. That's a proxy at best. No human eval, no error bars, no significance tests, no release of code or data (the paper says data_5k_GPT is a contribution but doesn't link it). The GPTScore circularity is real: GPT-generated data scored by an LLM judge likely inflates understandability/informativeness. Also, the accuracy aspect of GPTScore (ACC) is essentially classifying departments—BERT gets 0.996 F1, which is suspiciously high and suggests the task is easy or the classification is memorizing formatting. The paper acknowledges limitations (reversal curse, hallucination, department imbalance) in the conclusion, which is honest.\n\nIs it worth a serious referee? The dataset idea might be worth a workshop paper or a short paper if the authors re-frame it. But as written, the central claim is not supported. I'd desk-reject it in its current form, but I'd be open to a revised version that actually evaluates C-PATH's outputs against a baseline and releases the data. It's not incoherent—the authors clearly understand the pipeline—but it's not ready for publication as is.\n\nRecommendation: return to authors with the request to evaluate the actual system, add error bars, and release artifacts. If they do that, it could be a solid paper.","headline":"A genuinely useful GPT-based data generation idea is buried under a headline claim about a system that never actually gets evaluated.","tokens_in":13013,"tokens_out":3051,"would_cite":false,"duration_ms":24753,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fine-tuned 8-billion-parameter language model can run a natural conversation with a patient and recommend the right medical department.","keywords":["Large Language Models","Conversational AI","Patient Navigation","Medical Triage","Digital Health","Clinical Dialogue Systems","LLM fine-tuning","Synthetic dialogue generation"],"falsifier":"Run C-PATH on a set of standardized clinical vignettes with unscripted patient turns, and compare its department referral against a clinician panel or the DDXPlus ground truth; if the agreement is much lower than the 99% accuracy reported on data_5k_GPT, the proxy evaluation does not transfer to the actual conversation.","tokens_in":12025,"feed_emoji":"🩺","tokens_out":9370,"duration_ms":87328,"temperature":0.7,"pith_summary":"This paper claims that a small, open-source language model, fine-tuned in stages, can act as a conversational patient navigator: it asks about symptoms in plain language, keeps the thread over many turns, and ends by referring the patient to a specific medical department. The authors build C-PATH on the LLaMA3-8B base model by first injecting medical knowledge from three QA datasets, then tuning on doctor-patient dialogues, and finally tuning on clinical summaries. The key enabler, they argue, is a data-augmentation pipeline in which GPT-3.5 rewrites structured differential-diagnosis cases from DDXPlus into patient-friendly multi-turn conversations. On that GPT-rewritten dataset, a BERT classifier reaches nearly perfect department-recommendation accuracy, and GPTScore shows high understandability and informativeness. If the claims hold, an open, locally deployable triage assistant could reduce waiting-room pressure and simplify referrals without relying on closed commercial models.","feed_headline":"LLaMA3 chatbot triages patients to the right department","feed_subtitle":"Three-stage fine-tune plus GPT-rewritten medical dialogues yields near-perfect referral accuracy on the test set.","key_machinery":"The machinery that carries the argument is a three-stage fine-tuning pipeline on LLaMA3-8B: (1) knowledge injection with the PubmedQA, MedQA-USMLE, and MedMCQA question-answering corpora; (2) conversation tuning on doctor-patient dialogues, using a sliding-window history manager with optional turn-level summarization and explicit speaker tags so sessions can extend beyond the context limit; and (3) summary tuning on MTS-Dialog and ACI-BENCH pairs so the model can write EHR-ready notes. The second load-bearing element is the data-augmentation framework: GPT-3.5 rewrites each DDXPlus symptom question into several patient-friendly variants, diversifies yes/no responses, and even regenerates whole conversations, producing the data_5k_GPT corpus that the authors say normalizes style and keeps token counts low. These two pieces—the dataset recipe and the staged fine-tune—are what the paper credits for C-PATH's triage accuracy and dialogue quality.","core_discovery":"The central claim is that C-PATH, a conversational triage system built by fine-tuning LLaMA3-8B in three stages, can conduct multi-turn patient intake in lay language and recommend the appropriate medical department with high accuracy. The paper further claims that the dataset-construction method—having GPT-3.5 rewrite DDXPlus's structured symptom and pathology data into natural, varied doctor-patient dialogues—is what makes this work: it yields the strongest GPTScore results on understandability, informativeness, and patience burden, and it enables a standard BERT model to hit 99.6% accuracy on department prediction, surpassing ClinicalBERT. The authors also claim that the same fine-tuned model can generate structured EHR-style summaries of the conversation, so the triage dialogue feeds directly into clinical documentation. The paper presents this as evidence that task-specific fine-tuning on well-structured conversational data can beat domain-specific pretraining, and that LLM-based navigators are viable as supportive, not decision-making, tools.","pith_inferences":["The evaluation scores the rewritten datasets rather than C-PATH's own live responses, so a true test of the system would be to run real unscripted conversations and have clinicians rate the referrals.","The near-perfect BERT accuracy on data_5k_GPT may be inflated by the stylistic uniformity of GPT-generated text; real patients typing free-form descriptions are likely to be a harder input distribution.","The paper's conclusion that task-specific fine-tuning beats domain-specific pretraining rests on a single comparison between BERT and ClinicalBERT on one dataset family; a held-out corpus of real triage transcripts would make that claim more general.","A practical deployment would need a fallback protocol—the system currently makes a referral but does not flag emergencies or decide whether a patient should go to the emergency room versus a clinic."],"forward_implications":["An 8-billion-parameter open model, rather than a closed commercial API, could run triage conversations on hospital kiosks and mobile apps, keeping patient data on-premises.","The GPT-based rewriting recipe can be reused to turn any structured medical dataset into a dialogue-training corpus, reducing the need for scarce real doctor-patient transcripts.","The history-management strategy—pruning the window and summarizing older turns—lets multi-turn sessions run well past a model's token limit without losing the clinical thread.","If the summarization stage holds up, each triage conversation automatically yields an EHR-style note, cutting documentation time for intake staff.","The reported accuracy of the underlying dataset suggests that the bottleneck for such systems may be conversation data quality rather than model scale."],"supporting_citations":[{"why":"Supplies the structured differential-diagnosis cases with symptoms and pathologies that the pipeline rewrites into patient conversations.","marker":"[11]"},{"why":"Provides the LMFlow fine-tuning toolkit used in all three training stages and the evidence that a fully tuned LLaMA3-8B can beat ChatGPT on domain-specific tasks.","marker":"[8]"},{"why":"Defines the GPTScore instruction-based metric used to score the six dialogue-quality dimensions of the constructed datasets.","marker":"[13]"},{"why":"Contributes MedQA-USMLE questions for the medical knowledge injection stage.","marker":"[18]"},{"why":"Supplies PubMedQA instances for the knowledge injection stage; the paper also cites it for LLaMA3's biomedical QA strength.","marker":"[19]"},{"why":"Provides MedMCQA's large-scale medical multiple-choice data for the knowledge injection stage.","marker":"[32]"},{"why":"Gives doctor-patient conversation-summary pairs used to train the summarization stage.","marker":"[3]"},{"why":"Supplies ACI-BENCH conversation-note pairs for the summarization and EHR-note generation stage.","marker":"[47]"}],"fun_headline_variants":["LLaMA3 chatbot uses GPT-rewritten dialogues for triage","C-PATH: AI triage with GPT-crafted patient chats","GPT-rewritten dialogues train LLaMA3 for triage","Conversational AI triages patients with high accuracy","LLaMA3 triage model trained on GPT-3.5 dialogues"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's results assume that scoring the static, GPT-rewritten dialogue datasets with GPTScore and measuring BERT's department-prediction accuracy on those transcripts reflects how well C-PATH itself would triage real patients in a live conversation.","fun_headline_variants_meta":{"raw":{"variants":["LLaMA3 chatbot uses GPT-rewritten dialogues for triage","C-PATH: AI triage with GPT-crafted patient chats","GPT-rewritten dialogues train LLaMA3 for triage","Conversational AI triages patients with high accuracy","LLaMA3 triage model trained on GPT-3.5 dialogues"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000216,"raw_usage":{"total_tokens":1431,"prompt_tokens":946,"completion_tokens":485,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":395}},"tokens_in":562,"tokens_out":485,"duration_ms":4980,"temperature":1.0,"reasoning_tokens":395,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:50:21.121893+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run C-PATH on a set of standardized clinical vignettes with unscripted patient turns, and compare its department referral against a clinician panel or the DDXPlus ground truth; if the agreement is much lower than the 99% accuracy reported on data_5k_GPT, the proxy evaluation does not transfer to the actual conversation.","supporting_citations":[{"cited_title":"Ddxplus: A new dataset for automatic medical diag- nosis.Advances in Neural Information Processing Systems, 35:31306– 31318, 2022","cited_arxiv_id":null,"evidence_quote":"Supplies the structured differential-diagnosis cases with symptoms and pathologies that the pipeline rewrites into patient conversations."},{"cited_title":"LMFlow: An extensible toolkit for finetuning and inference of large foundation models","cited_arxiv_id":null,"evidence_quote":"Provides the LMFlow fine-tuning toolkit used in all three training stages and the evidence that a fully tuned LLaMA3-8B can beat ChatGPT on domain-specific tasks."},{"cited_title":"GPTScore: Evaluate as you desire","cited_arxiv_id":null,"evidence_quote":"Defines the GPTScore instruction-based metric used to score the six dialogue-quality dimensions of the constructed datasets."},{"cited_title":"Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering","cited_arxiv_id":null,"evidence_quote":"Provides MedMCQA's large-scale medical multiple-choice data for the knowledge injection stage."},{"cited_title":"An empirical study of clinical note generation from doctor-patient en- counters","cited_arxiv_id":null,"evidence_quote":"Gives doctor-patient conversation-summary pairs used to train the summarization stage."},{"cited_title":"Aci-bench: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation.Scientific data, 10(1):586, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies ACI-BENCH conversation-note pairs for the summarization and EHR-note generation stage."}],"review_version":1}