{"id":"fb9da35d-10d3-41e3-967f-c832031a19f5","arxiv_id":"2412.09946","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Fine-tuning GLM4 and LLaMA on a new Chinese nursing dataset improves F1 on nursing exams by a small margin, but the evaluation is at risk of training-test overlap.","lead":"The paper introduces a Chinese nursing dataset and fine-tunes large language models to improve their answers on nursing exam questions. It also sketches a chat-based nursing assistant for monitoring and care planning.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported nursing-exam gains may reflect memorization: NursingPiles is built from the same textbooks, manuals, and regulations as the benchmark (§3.2, §3.5), and no deduplication or contamination check is described.","rationale":"The reader's weakest assumption is exactly the load-bearing concern I identified: benchmark-test overlap with the training corpus, absent any deduplication. I read the paper in good faith and credit what is genuinely there: a new Chinese nursing corpus, a described IPT+SFT pipeline with concrete hyperparameters in Table 4, ablation results in Table 3 showing both components contribute, and a detailed ethics discussion. None of that, however, addresses the central evidential gap. The paper's headline result is a small performance edge over GPT-4o on a multiple-choice nursing exam. If the exam questions or close paraphrases appear in the 2.78M-token IPT corpus or in the 1.5M GPT-4o-generated dialogues, the fine-tuned model can improve by memorizing those items, and the comparison against GPT-4o becomes invalid as a test of generalization. The paper explicitly says the dataset is built from textbooks, manuals, regulations, and research papers, and the benchmark is drawn from the same professional examination literature; this makes overlap not merely possible but likely. The absence of any deduplication description, dataset release, or benchmark release prevents the reader from ruling out this explanation. I also note that the evaluation lacks statistical error bars, so even the small reported gap is not established as stable. My proposed test is concrete and decisive: compute overlap and re-evaluate on non-overlapping items. Because the reader already assigned a conditional verdict based on this concern, my stress test does not change the verdict; it reinforces it. If the overlap audit were run and came back clean, the central claim would be substantially strengthened, but that evidence is currently missing.","tokens_in":9086,"tokens_out":2695,"duration_ms":34108,"concrete_test":"Run an explicit overlap audit between every benchmark question in §3.5 and every item in NursingPiles (§3.2, Table 1). Normalize text by removing punctuation and whitespace, then compute exact-match and 10-gram Jaccard similarity for each benchmark question against all IPT documents, all single-turn SFT QA pairs, and all generated multi-turn dialogues. Flag any benchmark item with similarity above 0.8 to any training item. Then re-evaluate GLM4+IPT+SFT and GPT-4o on only the non-overlapping benchmark subset. If the F1/accuracy advantage over GPT-4o drops below the reported margins (0.91 F1 points, 2.06 accuracy points) or reverses, the strong claim in §4.1 is unsupported. Releasing the benchmark questions and the deduplicated dataset would make this check independently reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in §4.1 is that GLM4+IPT+SFT 'significantly outperforms' baseline models on the nursing exam benchmark. For that claim to hold as a statement of generalized nursing knowledge, the evaluation must not reward verbatim or near-verbatim retrieval of training items. Section 3.2 describes NursingPiles as assembled from textbooks, manuals, regulations, and research papers in the same domain, and Section 3.5 selects 'Three Basics and Three Stricts' and postgraduate nursing exam questions drawn from that same body of material. The paper never reports a deduplication or contamination filter between the IPT/SFT data and the benchmark, and it does not release the dataset or the evaluation questions. Because the reported advantage over GPT-4o is small (F1 86.21 vs 85.30; accuracy 58.9 vs 56.84), even a modest number of benchmark questions appearing verbatim or near-verbatim in the 2.78M IPT tokens, the 5,000 manual/regulation QA pairs, or the 1.5M GPT-4o-generated multi-turn dialogues could account for the entire gap. Table 1 specifically lists 'QA based on nursing safety and ethics from manuals, regulations' as SFT data, and the benchmark's 'Three Basics' questions cover the same manuals and regulation content, making overlap plausible rather than hypothetical. An additional, secondary risk is that the SFT data include dialogues generated by GPT-4o, which may itself have seen the benchmark questions; this could inflate the fine-tuning signal independently of direct overlap. The evaluation also lacks error bars or multiple-seed reporting, so the 0.91-point F1 gap over GPT-4o is not shown to be statistically robust. The contamination concern is the most load-bearing because it directly determines whether the paper's headline result measures nursing competence or training-set recall.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces NursingPiles, a Chinese nursing and elderly-care dataset assembled from textbooks, manuals, regulations, research papers, and GPT-4/GPT-4o-generated dialogues, and applies incremental pre-training (IPT) and supervised fine-tuning (SFT) to GLM4-Chat 9B and LLaMA 3.1 8B. The authors also describe a LangChain-based dynamic nursing assistant system. On a benchmark built from 'Three Basics and Three Stricts' and postgraduate nursing exam questions, the best system (GLM4 + IPT + SFT) reports Precision 86.78%, Recall 85.65%, F1 86.21%, and Accuracy 58.9%, slightly above GPT-4o (F1 85.30%, Accuracy 56.84%). The central empirical claim is that the IPT+SFT models significantly outperform the baselines.","tokens_in":9376,"tokens_out":4309,"duration_ms":49282,"significance":"If the reported results are valid, the paper offers a useful open-source Chinese nursing dataset, a reproducible training recipe for domain-adaptive fine-tuning, and a concrete application architecture for nursing and elderly care in Chinese settings. The ablation study is a sensible check on the contribution of each training stage, and the authors are transparent about several limitations, including the text-only focus and Chinese-only scope. However, the significance of the main claim depends on whether the benchmark measures generalized nursing knowledge rather than memorization of training material. The reported gains over GPT-4o are small, and the paper does not describe any contamination check between NursingPiles and the evaluation set, nor does it report statistical significance. These issues must be resolved before the improvements can be accepted as robust evidence of domain-adaptive capability.","major_comments":[{"comment":"The central claim in §4.1 that the integrated models 'significantly outperform' baselines assumes that the benchmark measures generalized nursing knowledge. The benchmark is built from 'Three Basics and Three Stricts' and postgraduate nursing exam questions (§3.5), while NursingPiles is assembled from textbooks, manuals, regulations, and research papers in the same domain (§3.2, Table 1). The paper does not report any deduplication or overlap filtering between NursingPiles and the evaluation questions. Because the reported gains over GPT-4o are small (F1 86.21 vs 85.30; Accuracy 58.9 vs 56.84), even a small number of verbatim or near-verbatim overlapping items among the 2.78M IPT tokens, the 5,000 manual/regulation QA pairs, or the 1.5M generated dialogues could account for the entire difference. To support the generalization claim, the authors should either describe and run an explicit contamination check between the benchmark and all training subsets and report the overlap rate, or re-evaluate on a disjoint held-out set.","section":"§3.2 and §3.5"},{"comment":"The paper states that the integrated models 'significantly outperform' baseline models, but no error bars, confidence intervals, or significance tests are provided, and the evaluation protocol is not fully specified. The number of multiple-choice versus open-ended questions is not stated, and it is not clear how Precision, Recall, F1, and Accuracy are computed for each question type or how open-ended responses are scored. The reported margins over GPT-4o are 0.91 F1 points and 2.06 accuracy points; without a variance estimate or significance test these differences could be noise. The authors should report the benchmark size and composition, define the scoring rules, and provide a bootstrap or paired permutation test, or multiple fine-tuning seeds, to justify the word 'significant.'","section":"§4.1, Table 2"},{"comment":"A secondary leakage path is not addressed: the multi-turn SFT data include 1M and 0.5M dialogues generated by GPT-4 and GPT-4o, and GPT-4o is itself a baseline in Table 2. If the generation process incorporated exam-like questions from the same nursing knowledge sources, the fine-tuned model could inherit benchmark information indirectly, even without direct dataset overlap. The paper should specify the prompts and source material used for the GPT-4/GPT-4o generation and state whether any benchmark-related filters or deduplication were applied to these generated dialogues.","section":"Table 1"}],"minor_comments":[{"comment":"The full-model rows contain '(–) (–) (–) (–)' as delta values; this is confusing because the column header says the parenthetical values represent the difference between the full model and the ablation variants. Please replace the dashes with explicit baseline values such as 0.00, or add a footnote clarifying that the full model is the reference.","section":"Table 3"},{"comment":"The terms 'Precision,' 'Recall,' 'F1,' and 'Accuracy' are used without formulas or a precise description of how they are aggregated over the benchmark questions; please add equations and a sentence stating the unit of analysis (e.g., per question, per answer token).","section":"§4.1, §4.2"},{"comment":"The text contains two unresolved placeholder links, 'this link' and 'link,' for the consent form and the ethics approval application. Please provide the actual URLs or state that they are available on request.","section":"Section 7"},{"comment":"There is an apparent inconsistency between §7.1.1, which states that participants were informed their facial images might appear in the dataset, and §7.2, which states that all images exclude recognizable facial features. Please clarify which statement is accurate.","section":"Section 7.2"},{"comment":"Some citations are incomplete or inconsistently formatted, such as 'Nurs Educ Perspect. 2023' and 'Perspect, 2023' in the related-work section, and the dataset is described as released but no data link is provided. Please correct these issues before publication.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The main concern for the editor is that the paper's key empirical claim depends on ruling out training/evaluation leakage, and the current manuscript does not describe any contamination check. The small margins over GPT-4o make this especially important. I would recommend asking the authors to add a leakage analysis and statistical significance tests before resubmission. The ethical statement inconsistency about facial images in Section 7 should also be resolved, as it is relevant to the journal's data-policy review."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a competent but standard fine-tuning pipeline for Chinese nursing, with a new dataset that would be useful if it were released. The problem is that the headline result—a ~2-point accuracy edge over GPT-4o on nursing exam questions—is not interpretable as generalized knowledge because the paper never checks whether those exam questions already appear in the training corpus.\n\nWhat the paper does well: the NursingPiles dataset is thoughtfully assembled from textbooks, manuals, regulations, and simulated dialogues, and the ethics section is more detailed than most. The ablation study is a sensible idea, and the authors correctly identify that both IPT and SFT contribute. Those are real contributions, though the dataset is not actually released.\n\nThe soft spots are load-bearing. Contamination is the main one. NursingPiles includes textbooks, manuals, and regulations from the same domain as the 'Three Basics and Three Stricts' exam, and there is no deduplication or leakage analysis anywhere. With an F1 gap of only 0.91 points over GPT-4o, a handful of verbatim benchmark questions in the training data would fully explain the result. The paper also reports 'significant' improvements without any error bars or significance tests—that word is doing too much work. Table 3 literally has placeholder dashes in the full-model rows where deltas should go. The comparison omits LlamaCare and Xiong et al., both cited as related work, so 'SOTA' and 'pioneered' are not supportable claims.\n\nThe citation pattern is fine overall, but the evaluation protocol is too vague to substantiate the central claim. The right fix is clear: release the dataset, run an explicit overlap analysis between train and test, add multiple seeds or at least some variance reporting, and benchmark against prior nursing models.\n\nThis paper is for readers in applied nursing informatics who want a Chinese-language nursing resource—but right now it is a dataset-paper-in-waiting, not a demonstrated result. I would send it to peer review with a request for major revision, because the dataset and ablation have value and the contamination issue is addressable. Without those changes, the paper's claims do not hold.","headline":"Useful dataset, unproven result: the small GPT-4o edge may just be training-set recall, not nursing competence.","tokens_in":10033,"tokens_out":2358,"would_cite":false,"duration_ms":27993,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that incremental pretraining plus supervised fine-tuning on a Chinese nursing corpus produces an open-weight LLM that slightly outperforms GPT-4o on a Chinese nursing exam benchmark.","keywords":["large language models","nursing","elderly care","incremental pretraining","supervised fine-tuning","Chinese nursing dataset","NursingPiles","LLM benchmark"],"falsifier":"Compare every benchmark exam question, or a near-duplicate paraphrase, against NursingPiles using exact match and n-gram or embedding similarity; if a meaningful fraction of test items appear in the training data, the outperformance over GPT-4o could be memorization rather than generalized nursing competence.","tokens_in":8849,"feed_emoji":"🩺","tokens_out":7968,"duration_ms":77547,"temperature":0.7,"pith_summary":"This paper argues that an open-weight language model can be turned into a nursing-domain specialist by adding two stages on top of the base model: incremental pretraining on Chinese nursing texts and supervised fine-tuning on nursing question-answer pairs. It builds a multi-source Chinese nursing corpus, NursingPiles, and evaluates the tuned models on Chinese nursing exam questions. The best model, GLM4-Chat 9B with both stages, reaches 86.2% F1 and 58.9% accuracy, slightly above GPT-4o's 85.3% F1 and 56.84% accuracy. The paper also sketches a LangChain-based nursing assistant intended to monitor patients and generate personalized care plans. If the result holds, it would show that a relatively small, open, domain-adapted model can outperform a frontier general-purpose model on a narrow clinical knowledge test.","feed_headline":"Nursing-tuned model edges out GPT-4o on Chinese exam","feed_subtitle":"Fine-tuning on a new Chinese nursing corpus pushes a 9B open model to 86.2% F1, above GPT-4o's 85.3%.","key_machinery":"The load-bearing mechanism is the two-stage training recipe applied to a base LLM. Incremental pretraining (IPT) first exposes the model to 2.78 million tokens of Chinese nursing textbooks and manuals; supervised fine-tuning (SFT) then trains it on 17,580 single-turn QA pairs, 5,000 nursing safety and ethics QA pairs, 5,000 medical open-source QA pairs, and roughly 1.5 million GPT-generated multi-turn dialogues, all delivered through low-rank adaptation (LoRA). The evaluation machinery is a set of Chinese nursing exam questions from the Three Basics and Three Stricts test and the postgraduate nursing exam, scored by precision, recall, F1, and accuracy. The LangChain components are application-level glue: data collection, diagnosis triggers, care-plan generation, and follow-up chains, but they do not enter the benchmark numbers.","core_discovery":"On its own terms, the paper's central claim is that an open-weight large language model can be specialized to nursing by a two-stage recipe: incremental pretraining (IPT) on a Chinese nursing corpus, followed by supervised fine-tuning (SFT) on nursing QA pairs. The flagship result is GLM4-Chat 9B with IPT+SFT scoring a precision of 86.78%, recall of 85.65%, F1 of 86.21%, and accuracy of 58.9% on a Chinese nursing exam benchmark, topping GPT-4o's 86.62/84.02/85.3/56.84. The paper also claims both stages matter: dropping either IPT or SFT lowers F1 and accuracy for both base models. It introduces NursingPiles, a Chinese nursing corpus spanning textbooks, manuals, regulations, research papers, and simulated dialogues, and presents a LangChain-based nursing assistant for monitoring, diagnosis, and care-plan generation as the application layer.","pith_inferences":["If exam questions overlap with NursingPiles, the reported gains shrink or vanish; the paper does not describe any contamination screening between the test set and the training corpus.","The accuracy gap (58.9% versus F1 86.2%) hints that the model is much better at ranking plausible options than at committing to the single correct answer, a distinction that matters for real care decisions.","A direct test of the framework's generality would be to apply the same IPT+SFT recipe to a nursing exam written after the corpus cutoff, or to an out-of-domain clinical reasoning benchmark.","The privacy and security components are architectural sketches; nothing in the experiments measures whether the assistant protects patient data or avoids biased advice in live use."],"forward_implications":["A 9-billion-parameter open-weight model tuned on domain-specific Chinese nursing data can beat a much larger frontier model on a nursing knowledge exam, which lowers the cost barrier for deploying specialized clinical assistants.","The ablation results imply that neither IPT nor SFT alone is enough; the full protocol is the portable unit of the paper's method.","The NursingPiles construction pipeline, including synthetic multi-turn dialogues, offers a template for building domain corpora in other specialties where public data are thin.","The benchmark numbers are only for multiple-choice knowledge; the paper does not claim that the assistant's care plans or diagnoses were clinically validated."],"supporting_citations":[{"why":"Provides the GLM4-Chat 9B base model that yields the best results after IPT and SFT.","marker":"GLM et al., 2024"},{"why":"Provides LLaMA 3.1, the second base model used in the comparison and ablations.","marker":"Vavekanand and Sam, 2024"},{"why":"Source of the Three Basics and Three Stricts exam questions used in the benchmark.","marker":"Zhang, 2020"},{"why":"Source of the postgraduate nursing exam questions used in the benchmark.","marker":"Li, 2019"},{"why":"LlamaCare is the closest prior instruction-tuned clinical LLM the paper positions itself against.","marker":"Li et al., 2024"},{"why":"Motivates the open-source data-mixing strategy that guards against catastrophic forgetting during fine-tuning.","marker":"Zhai et al., 2023"},{"why":"MedNgage supplies a prior patient-nurse conversation dataset that the paper contrasts with NursingPiles.","marker":"Wang et al., 2023b"},{"why":"Prior nursing decision-making LLM with local knowledge bases, cited as related text-only work.","marker":"Xiong et al., 2023"}],"fun_headline_variants":["9B model tops GPT-4o on Chinese nursing exam","New nursing corpus helps small LLM beat GPT-4o","Incremental training lifts 9B LLM past GPT-4o in nursing","Chinese nursing dataset fine-tunes 9B model beyond GPT-4o","9B LLM with nursing tuning overtakes GPT-4o"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation scores assume the benchmark exam questions were not present in the nursing textbooks, manuals, and regulations used to build NursingPiles, but no contamination check or deduplication between the training corpus and the test set is described.","fun_headline_variants_meta":{"raw":{"variants":["9B model tops GPT-4o on Chinese nursing exam","New nursing corpus helps small LLM beat GPT-4o","Incremental training lifts 9B LLM past GPT-4o in nursing","Chinese nursing dataset fine-tunes 9B model beyond GPT-4o","9B LLM with nursing tuning overtakes GPT-4o"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000912,"raw_usage":{"total_tokens":3866,"prompt_tokens":838,"completion_tokens":3028,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":2933}},"tokens_in":454,"tokens_out":3028,"duration_ms":22785,"temperature":1.0,"reasoning_tokens":2933,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:31:36.143240+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare every benchmark exam question, or a near-duplicate paraphrase, against NursingPiles using exact match and n-gram or embedding similarity; if a meaningful fraction of test items appear in the training data, the outperformance over GPT-4o could be memorization rather than generalized nursing competence.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides LLaMA 3.1, the second base model used in the comparison and ablations."},{"cited_title":"three basics and three stricts","cited_arxiv_id":null,"evidence_quote":"Source of the Three Basics and Three Stricts exam questions used in the benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior nursing decision-making LLM with local knowledge bases, cited as related text-only work."}],"review_version":1}