{"id":"ea059d32-612b-4d99-8e09-be2138c34154","arxiv_id":"2508.15192","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"LLM4Sweat reports high accuracy on hyperhidrosis MCQ tasks after fine-tuning on synthetic data generated from the test set, conflating memorization with generalization.","lead":"This paper fine-tunes small open-source language models on AI-generated hyperhidrosis questions, claiming they outperform general-purpose baselines in diagnosis and treatment advice. The evaluation is not credible because the training questions were generated from the test questions, and the 'expert' reviewers were also an AI.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Synthetic training data are generated 'based on the real testing dataset' (Experiments), so the 0.925/0.925 accuracies may reflect test-set leakage rather than generalization; the central claim is not supported as reported.","rationale":"The reader's weakest_assumption identified exactly the same load-bearing concern: the synthetic training vignettes are generated from the real testing dataset, breaking independence between training and evaluation. My reading of the full text confirms this is not a minor limitation but a structural flaw in the experimental design. The paper explicitly states the synthetic data are 'based on the real testing dataset,' and the 'expert' validation is performed by a frontier LLM rather than human specialists, contradicting the abstract's claim of clinical and psychological specialist assessment. Because the central claim of superior performance is derived entirely from this contaminated setup, the verdict of REJECT is appropriate. The proposed concrete test would settle whether the leakage actually inflates the numbers; if the re-run with proper separation still shows large gains, the concern would be mitigated, but as written the evidence does not support the claim.","tokens_in":9227,"tokens_out":1812,"duration_ms":20286,"concrete_test":"Re-run the fine-tuning pipeline with a proper data split: hold out a randomly selected 20 of the 80 real questions as the test set, generate the 180 synthetic vignettes only from the remaining 60 real questions (and other sources explicitly not containing the held-out questions), then fine-tune the same Llama-3.2-1B and 3B models and evaluate on the held-out 20. If the resulting accuracy is materially lower than 0.925/0.875 diagnosis and 0.825/0.925 treatment, the reported numbers in Table 1 are attributable to test-set leakage. A complementary check is to generate synthetic training data de novo from clinical guidelines (e.g., IHHS, Mayo Clinic) without referencing the 80-question test set and re-evaluate on the original 80 questions; if performance drops, the original setup overfit the test distribution.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central empirical claim—that LLM4Sweat substantially outperforms baselines—rests on an evaluation whose training distribution is explicitly derived from the test distribution. In the Experiments section, the authors state: 'we generated 180 synthetic vignettes (90 diagnosis, 90 treatment) using a frontier LLM (GPT-5) based on the real testing dataset.' Because the 80-question testing benchmark is the sole source of the synthetic vignettes used for fine-tuning, the model is trained on paraphrases and variants of the exact questions it is later tested on. This is a direct train/test contamination that inflates all reported accuracy, precision, recall, and F1 figures. The independent support claimed by the ablation studies is also compromised: the 'expert evaluation' loop is not performed by human specialists but by the same frontier LLM ('In this work, it is done by the frontier LLM'), so the iterative enrichment likewise depends on the test-derived distribution. Without a clean separation between synthetic training generation and evaluation, the numbers in Table 1 cannot be interpreted as evidence of generalization to unseen hyperhidrosis queries, breaking the paper's central claim and its stated potential for rare-disease transfer.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LLM4Sweat, a three-stage pipeline for adapting small open-source LLMs (Llama-3.2-1B/3B) to hyperhidrosis diagnosis, treatment recommendation, and psychological support. Stage 1 uses a frontier LLM to generate synthetic vignettes from curated open-source data; Stage 2 fine-tunes the base model on those vignettes; Stage 3 adds an 'expert-in-the-loop' step in which outputs are reviewed and validated responses are fed back for further fine-tuning. The authors report that LLM4Sweat substantially outperforms unadapted baselines, with 1B and 3B models reaching 0.925 accuracy in diagnosis and treatment respectively, and claim the framework is transferable to other rare diseases. The central empirical claim, however, is compromised by a direct train/test overlap: the synthetic training vignettes are explicitly generated 'based on the real testing dataset,' and the 'expert' validation is performed by the same frontier LLM rather than by human specialists.","tokens_in":9583,"tokens_out":3826,"duration_ms":46875,"significance":"If the reported results were valid, the paper would offer a useful blueprint for adapting small LLMs to rare medical domains with scarce data, and the emphasis on integrated diagnosis, treatment, and psychological support is sensible. The ablation structure (with/without expert evaluation) is a reasonable design. However, the empirical contribution is not established. The training data are generated from the test data, and the 'expert' feedback is simulated by an LLM using the same test-derived queries. Consequently, the accuracy numbers in Table 1 cannot be interpreted as evidence of generalization, and the claimed trustworthiness is not supported by human evaluation. The paper also does not provide code, data, or a repository despite describing the framework as open-source. The domain motivation and problem framing are timely, but the central claims require clean, independent evaluation before the results can be accepted.","major_comments":[{"comment":"D_real is defined in Eq. (1) as the testing dataset, and the synthetic training set is described as '180 synthetic vignettes ... using a frontier LLM (GPT-5) based on the real testing dataset.' Since Eq. (3) trains on D_syn and Table 1 evaluates on D_real, the fine-tuned model is trained on a distribution explicitly derived from the test distribution. This is train/test contamination, not augmentation from independent open-source data. The reported accuracy, precision, recall, and F1 values in Table 1 are therefore inflated and cannot support the central claim that LLM4Sweat 'substantially outperforms baseline models.'","section":"Experiments, Datasets (Synthetic training dataset); Eq. (1)-(3)"},{"comment":"The expert-in-the-loop component is not human expert evaluation. The manuscript states 'In this work, it is done by the frontier LLM' and 'we used the state-of-the-art LLM to represent specialists for practice.' The 40 'validated responses' added to the training set are generated from the same test queries. The ablation comparing models with and without expert evaluation therefore measures additional fine-tuning on test-derived outputs, not the effect of genuine expert validation. This invalidates the claimed trustworthiness and the conclusion that 'expert-in-the-loop refinement provides the final layer of trustworthiness.'","section":"Inference and Expert Evaluation Stage; Path to Deployment"},{"comment":"The evaluation uses only 80 multiple-choice questions (40 per task), with no separate validation set. Hyperparameters (learning rate from {5e-6, 5e-5, 2e-4, 1e-3}; epochs from {1,3,5}) are tuned on the same test set, and no confidence intervals or significance tests are reported. Given the small test size, a change of a few questions moves accuracy by 0.025, so the presented differences (e.g., 0.825 vs 0.925) are not shown to be reliable. A clean held-out test set that is independent of all training and validation decisions is required to underwrite the performance claims.","section":"Experiments, Hyperparameter Settings and Results"},{"comment":"The limitations section acknowledges the small 80-question test set and the need for future human evaluation, but it does not acknowledge the core circularity that the synthetic training set is generated from the test set. Consequently, the abstract and conclusions overclaim: the paper states it 'delivers the first open-source LLM framework for hyperhidrosis' and offers 'a generalizable approach for other rare diseases,' but no independent evidence of generalization is presented.","section":"Discussions and Limitations; Conclusions"}],"minor_comments":[{"comment":"Typo: 'vigenette' should be 'vignette'.","section":"Methodology, Eq. (2)"},{"comment":"Figure 2 is dense and contains in-panel example prompts and answers that distract from the workflow; the caption should clearly state that the 'expertise enrichment' is simulated by the frontier LLM in this work, not by human specialists.","section":"Figure 2"},{"comment":"The rows labeled 'w/o Expert Eval' are essentially ablation conditions; the caption should clarify that these are fine-tuned on the synthetic set only, while the 'LLM4Sweat' rows include additional test-derived validated responses.","section":"Table 1"},{"comment":"The paper claims to present an 'open-source' framework but does not provide a repository, dataset, or model weights. Adding a link or clear availability statement would be necessary for reproducibility.","section":"General"},{"comment":"Some references (e.g., Barr et al. 2025, Zhao et al. 2025) are cited with incomplete bibliographic information; please supply full venue and identifiers.","section":"Related Work"}],"recommendation":"reject","confidential_remarks":"The core circularity is a load-bearing flaw: the training set is generated from the test set and the 'expert' evaluation is simulated by the same frontier LLM using the same test queries. This cannot be fixed by local revision; the experiments must be redesigned with a truly independent test set and human expert validation. If the authors are willing to re-run the full pipeline under those conditions, the framework idea could be of interest, but the current manuscript does not support its empirical claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First: the new thing here is the application, not the method. LLM4Sweat is the first hyperhidrosis-tailored LLM, and the authors assemble a small 80-question multiple-choice benchmark from reputable sources. That benchmark has some standalone value, and the framing—small open models plus synthetic data for a rare disease—is a sensible problem to work on.\n\nThe writing is clear and the pipeline is standard: frontier LLM generates synthetic vignettes, LoRA fine-tuning, then an \"expert\" loop. The ablation design is reasonable in isolation.\n\nThe soft spot is not minor. The synthetic training set is generated \"based on the real testing dataset\" (their words, in Experiments). That means the model is fine-tuned on paraphrases and variants of the exact questions it is later tested on. The reported gains—0.925 diagnosis accuracy for the 1B model, 0.925 treatment for the 3B—are therefore likely a measure of memorizing the test distribution, not of generalizing to unseen hyperhidrosis queries. The expert loop compounds the problem: it is not performed by clinical specialists as the abstract implies, but by the same frontier LLM, using the same test queries. That is not an independent assessment.\n\nThere are smaller issues. The counseling task, listed as one of the three integrated tasks, gets no evaluation at all. No code or data are released despite the \"open-source\" label. There are no error bars or significance tests, which matters less given the main flaw but still weakens the presentation.\n\nOn balance: the central empirical claim does not hold as reported. The paper is salvageable only if the experiments are redone with synthetic data generated independently of the evaluation set—e.g., from the real open-source sources only—and with human expert review of a held-out set. The benchmark itself could be reused.\n\nWho should read this? Anyone working on low-resource medical LLM fine-tuning, mostly as a cautionary example of how easy it is to leak test information through synthetic data. It deserves a serious referee because the application is relevant and the fix is concrete, but it should not be published in its current form.","headline":"The paper's central claim is voided by train/test leakage: synthetic training vignettes are explicitly generated from the test set, so the 0.925 accuracies can't be read as generalization.","tokens_in":9983,"tokens_out":2495,"would_cite":false,"duration_ms":28432,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning on synthetic vignettes takes small open-source LLMs from 0.425 to 0.875 overall accuracy on hyperhidrosis support.","keywords":["hyperhidrosis","large language models","synthetic data augmentation","rare disease","fine-tuning","expert-in-the-loop evaluation","clinical NLP","low-resource medical AI"],"falsifier":"Build a held-out set of 80 hyperhidrosis multiple-choice questions written by dermatologists after the synthetic vignettes were generated, with no overlap in wording, and rerun the fine-tuned models. If accuracy on those fresh questions falls far below 0.875-0.900 while accuracy on the current test set stays high, the reported gains come from training on cases derived from the test set, not from learned domain reasoning.","tokens_in":9144,"feed_emoji":"💧","tokens_out":6619,"duration_ms":69944,"temperature":0.7,"pith_summary":"LLM4Sweat tries to show that data scarcity for rare diseases can be bypassed by having a frontier LLM generate synthetic training vignettes from sparse curated sources. The paper fine-tunes two small open-source models (1B and 3B) on those vignettes plus expert-validated outputs, and reports accuracy gains from 0.425 to 0.875 (1B) and from 0.588 to 0.900 (3B) on an 80-question hyperhidrosis benchmark. Diagnosis reaches 0.925 for the 1B model and treatment reaches 0.925 for the 3B model, with F1 scores above 0.85. If the result holds on genuinely unseen data, it would give a transferable blueprint for building trustworthy LLM assistants for other rare conditions where real data are too sparse to fine-tune general models.","feed_headline":"Small AI models hit ~90% accuracy on hyperhidrosis support","feed_subtitle":"Synthetic vignettes from a frontier LLM lift 1B/3B models from 0.425 to 0.875 overall accuracy.","key_machinery":"The central mechanism is the three-stage closed loop: (1) data augmentation by a frontier LLM that converts sparse curated open-source material into balanced synthetic question-answer vignettes; (2) LoRA-based supervised fine-tuning of an open-source base model on those vignettes; (3) expert-in-the-loop inference, where specialist validation of model outputs produces additional curated pairs that are fed back into the training data. The repeated feedback of validated responses is what separates the framework from a one-shot synthetic-data fine-tune and yields the incremental accuracy gains in the ablation.","core_discovery":"On its own terms, the paper establishes that a small, open-source language model can be made clinically useful for a rare condition without large real-world datasets. The reported mechanism is a closed loop: a frontier LLM turns curated open-source material into 180 balanced synthetic vignettes; the base model is fine-tuned on them; at inference, evaluated responses from specialists are fed back into the training pool. Measured on an 80-question multiple-choice benchmark built from clinician-facing sources, this lifts the 1B model from 0.425 to 0.875 overall accuracy and the 3B model from 0.588 to 0.900, with best per-task accuracy of 0.925 in diagnosis (1B) and 0.925 in treatment (3B). The","pith_inferences":["A reader should treat the 0.875-0.900 accuracy figures as upper-bound estimates until the models are evaluated on a held-out set written independently of the synthetic training vignettes; the paper states the synthetic vignettes were generated from the real testing dataset, and that overlap is the main uncontrolled risk.","Because this experiment used a frontier LLM to stand in for clinical specialists, the reported expert-evaluation gain may partly reflect the generator's own consistency. Real clinician review could either strengthen or weaken that increment.","A natural extension would be to run the same pipeline on another rare dermatological condition with public clinical guidelines, using a fresh human-authored test set to separate the data-augmentation effect from test-set leakage."],"forward_implications":["If accurate, the same pipeline could be pointed at any rare disease with a small amount of trustworthy source material and produce a deployable assistant without expensive data collection.","The 1B/3B size of the fine-tuned models means the system could run on modest hardware, making clinic- or patient-facing deployment feasible in resource-constrained settings.","The combination of synthetic augmentation and expert-in-the-loop refinement suggests a way to keep improving accuracy after deployment, since real validated interactions continuously enrich the training set.","The consistent gains across two model sizes make the improvement attributable to the framework rather than to a lucky choice of base model."],"supporting_citations":[{"why":"Establishes that LLM-generated synthetic clinical text can improve low-resource NLP, the premise for the data augmentation stage.","marker":"[Barr et al. 2025]"},{"why":"Supports the use of generative models to expand rare-disease datasets before fine-tuning.","marker":"[Kazeminia et al. 2020]"},{"why":"Shows LLMs encode clinical knowledge, motivating the choice to fine-tune rather than build a system from scratch.","marker":"[Singhal et al. 2023]"},{"why":"Provides the clinical diagnostic reasoning that the test questions evaluate.","marker":"[Henning et al. 2021]"},{"why":"Provides the treatment ladder (topicals to procedural options) reflected in treatment questions.","marker":"[Hong, Lupin, and O'Shaughnessy 2012]"},{"why":"Documents hyperhidrosis prevalence and burden, motivating the clinical need.","marker":"[Strutton et al. 2004]"}],"fun_headline_variants":["Synthetic vignettes lift small LLMs to ~90% hyperhidrosis accuracy","Open-source LLM for hyperhidrosis hits 90% via synthetic data","LLM4Sweat: small model, 90% accuracy for excessive sweating","Small AI, big accuracy: synthetic data powers hyperhidrosis support","From rare disease to AI support: synthetic vignettes fine-tune small LLMs"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The reported gains depend on the synthetic training cases not being near-copies of the test questions; since they were generated from the test set, any overlap would inflate accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic vignettes lift small LLMs to ~90% hyperhidrosis accuracy","Open-source LLM for hyperhidrosis hits 90% via synthetic data","LLM4Sweat: small model, 90% accuracy for excessive sweating","Small AI, big accuracy: synthetic data powers hyperhidrosis support","From rare disease to AI support: synthetic vignettes fine-tune small LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001093,"raw_usage":{"total_tokens":4430,"prompt_tokens":801,"completion_tokens":3629,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":3526}},"tokens_in":545,"tokens_out":3629,"duration_ms":32199,"temperature":1.0,"reasoning_tokens":3526,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T18:02:07.036717+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a held-out set of 80 hyperhidrosis multiple-choice questions written by dermatologists after the synthetic vignettes were generated, with no overlap in wording, and rerun the fine-tuned models. If accuracy on those fresh questions falls far below 0.875-0.900 while accuracy on the current test set stays high, the reported gains come from training on cases derived from the test set, not from learned domain reasoning.","supporting_citations":[{"cited_title":"A.; Quan, J.; Guo, E.; and Sezgin, E","cited_arxiv_id":null,"evidence_quote":"Establishes that LLM-generated synthetic clinical text can improve low-resource NLP, the premise for the data augmentation stage."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the use of generative models to expand rare-disease datasets before fine-tuning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows LLMs encode clinical knowledge, motivating the choice to fine-tune rather than build a system from scratch."},{"cited_title":"A.; Thorlacius, L.; Ibler, K","cited_arxiv_id":null,"evidence_quote":"Provides the clinical diagnostic reasoning that the test questions evaluate."},{"cited_title":"H.; Lupin, M.; and O'Shaughnessy, K","cited_arxiv_id":null,"evidence_quote":"Provides the treatment ladder (topicals to procedural options) reflected in treatment questions."},{"cited_title":"R.; Kowalski, J","cited_arxiv_id":null,"evidence_quote":"Documents hyperhidrosis prevalence and burden, motivating the clinical need."}],"review_version":1}