{"id":"85e6376a-db5e-45fa-920e-2d57541fc717","arxiv_id":"2509.05867","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"A GraphRAG-plus-fine-tuning TCM formula generator outperforms baselines in its own experiments, but its theoretical proofs and evaluation are not sound.","lead":"This paper combines graph-based retrieval with fine-tuned language models to generate Traditional Chinese Medicine formulas with added explanations such as herb roles, contraindications, and pulse diagnosis. The authors claim mathematical proofs that this combination reduces errors and hallucinations, but those proofs contain invalid steps and the evaluation leans on model-generated references.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 1's claimed bound does not follow: the proof switches from γ²/(2β) to γ/β after 'integrating over T steps', and the gradient-norm lower bound by mutual information is asserted without justification.","rationale":"The reader's verdict is REJECT, and I find the same central weakness but locate it slightly differently. The reader's weakest_assumption points at the DeepSeek-generated ground-truth references used for SFT, DPO, and BLEU/ROUGE evaluation; that is a real concern about circularity. However, the more load-bearing issue is the invalidity of Proposition 1, because the abstract explicitly advertises 'novel theoretical proofs demonstrating that integrating GraphRAG with fine-tuning techniques can reduce generalization error and hallucination rates.' If that proof is invalid, the paper's primary claimed contribution fails even before the evaluation is considered. The proof's fatal step is the unproven transition from a per-step gradient-descent bound to the final T-independent γ/β inequality, with the intermediate γ²/(2β) form. This is not a matter of presentation or missing details; the claimed inequality is not a logical consequence of the stated assumptions. Proposition 2's uniform-preference step and Propositions 3–4's event manipulations are also flawed, but Proposition 1 alone already breaks the central theorem. The empirical tables may still indicate practical gains, but they cannot substitute for the stated proof, especially since the evaluation references are model-generated. A single concrete check, re-deriving the inequality chain with an explicit summation, settles whether the proof can be repaired. I therefore recommend no change to the reader's REJECT verdict.","tokens_in":30721,"tokens_out":4453,"duration_ms":39766,"concrete_test":"Re-derive the chain in §III-F.1 by replacing 'integrate over T steps' with an explicit summation. Check whether the inequalities imply E(θ_T) ≤ E(θ_0) − Σ_{t=1}^T ||∇E(θ_t)||²/(2β), and whether the assertion ||∇E(θ_t)||² ≥ I_t² can be proved from the definition of I. Determine whether the final bound E(θ_GraphRAG+SFT) ≤ E(θ_SFT) − γ/β follows for any T, γ, β; for example, take T=1, γ=β=1, which makes the claimed RHS E_SFT − 1 while the proven per-step bound is at best E_SFT − 1/2. If the claimed bound cannot be instantiated, Proposition 1 is false as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that GraphRAG+SFT provably reduces generalization error (Eq. 3). The proof in §III-F.1 derives, under β-smoothness and step size η=1/β, the per-step decrease E(θ_{t+1}) ≤ E(θ_t) − ||∇E(θ_t)||²/(2β). It then asserts without derivation that ||∇E(θ_t)||² ≥ I(y;c|x)², where I is conditional mutual information between retrieval context c and target y. No link between the loss gradient and the mutual information of the conditional distribution is established; this is an unproved assumption, not a consequence of the definitions. Even granting that inequality, the text substitutes θ_GraphRAG+SFT and θ_SFT for θ_{t+1} and θ_t and obtains E ≤ E_SFT − γ²/(2β). It then states, 'After integrating over T optimization steps', E ≤ E_SFT − γ/β. The exponent of γ changes from γ² to γ, the factor of 2 disappears, and the T-dependence vanishes, with no derivation. Summing the per-step bound over T steps would give E(θ_T) ≤ E(θ_0) − (1/(2β)) Σ_t ||∇E(θ_t)||², which cannot yield a T-independent γ/β term unless additional assumptions are supplied. Thus the proof of the headline 'provable reduction' claim is internally invalid; the claimed theorem is unsupported. This is load-bearing because the abstract and contributions state the theoretical reduction as a key result, independent of the empirical tables.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ZhiFangDanTai, a framework that combines Graph-based Retrieval-Augmented Generation (GraphRAG) with supervised fine-tuning (SFT) and direct preference optimization (DPO) for Traditional Chinese Medicine formula generation. The authors construct a fine-grained TCM instruction dataset from 80,000 collected records, use DeepSeek with 50 human-annotated seeds to generate detailed explanations, build a TCM knowledge graph, and fine-tune Llama3.2-7B. They claim novel theoretical proofs showing that GraphRAG plus fine-tuning reduces generalization error and hallucination rates, and they report large improvements over several baselines on both collected and clinical datasets using BLEU/ROUGE plus six TCM-oriented metrics. The paper also reports ablations over top-k, base models, and the inclusion of GPT, and it releases the model and dataset.","tokens_in":31120,"tokens_out":5211,"duration_ms":46084,"significance":"If the claims were correct, the paper would be a practically useful contribution to domain-specific medical LLM systems: it combines structured retrieval with fine-tuning, evaluates on a real clinical dataset in a zero-shot setting, and releases both the model and the instruction dataset. The authors also make an unusual and welcome effort to design TCM-specific evaluation metrics (CCR, CSCR, CCHR, FS, SCR, LR) and to ablate major design choices. However, the two central pillars of the paper are not currently supported. The theoretical section contains multiple unjustified or internally inconsistent steps, including the key Proposition 1, and the empirical evaluation has a closed loop in which DeepSeek-generated outputs serve simultaneously as training targets, preference sources, and evaluation references. As a result, the headline claims of provable generalization improvement and reduced hallucination are not established by the evidence presented.","major_comments":[{"comment":"The claimed reduction E(θ_GraphRAG+SFT) ≤ E(θ_SFT) − γ/β does not follow from the proof. The derivation first obtains the per-step bound E(θ_{t+1}) ≤ E(θ_t) − ||∇E(θ_t)||²/(2β), and then, even granting the unproved lower bound ||∇E(θ_t)||² ≥ I(y;c|x)², gives E(θ_{t+1}) ≤ E(θ_t) − γ²/(2β). The text then states that 'after integrating over T optimization steps' this becomes E(θ_GraphRAG+SFT) ≤ E(θ_SFT) − γ/β. This changes the exponent of γ from 2 to 1, drops the factor 1/2, and removes the T-dependence without any derivation. Summing the per-step bound under the stated assumptions would give E(θ_T) ≤ E(θ_0) − Tγ²/(2β), not the claimed expression. Because Proposition 1 is the formal basis for the abstract and contributions, this is a load-bearing error.","section":"§III-F.1, Proposition 1"},{"comment":"The assertion ∥∇E(θ_t)∥² ≥ I(y;c|x)² is introduced without proof. The conditional mutual information I(y;c|x) is a property of the data distribution (x,c,y), whereas ∇E(θ_t) is a property of the loss landscape at a particular parameter vector; no link between the two is established. Earlier in the same proof, I(y;c|x) is used as a distributional quantity in the decomposition E[−log P(y|x,c)] = E[−log P(y|x)] − I(y;c|x), while E(θ_SFT) is later interpreted as an expected negative log-likelihood of the model Pθ(y|x,c). The proof thus conflates the true data distribution with the model distribution and effectively assumes the quantity it needs to prove. This gap cannot be filled from standard β-smoothness or step-size arguments.","section":"§III-F.1, Proposition 1 proof"},{"comment":"The derivation of E(θ_DPO) ≤ E(θ_SFT) − E[∆]/β contains several unjustified steps. The inequality β log(Pθ(yw|x,c)/Pref(yw|x,c)) − β log(Pθ(yl|x,c)/Pref(yl|x,c)) ≥ ∆ is asserted as the DPO optimization objective, but the DPO objective maximizes a log-sigmoid of that expression and no such lower bound follows. The subsequent manipulation uses 'DPO decreases the likelihood of y_l' to write Pref(yl)/Pθ(yl) ≥ 1 and then, in the next paragraph, uses 'DPO increases the likelihood of y_w' to obtain the opposite direction; both cannot be used as independent assumptions in the same bound. The 'uniform preference assumption' E[I(y=y_w)] = E[I(y=y_l)] = 0.5 is also stated without justification, and the final sign of E[∆] is never reconciled with ∆ being a positive log-likelihood ratio. Proposition 2 therefore does not constitute a valid proof of the stated error reduction.","section":"§III-F.2, Proposition 2"},{"comment":"The evaluation loop is closed. The ground-truth detailed explanations are produced by DeepSeek via in-context learning from only 50 human-annotated seeds, and the same DeepSeek-generated content is used simultaneously as the SFT target, as the source of DPO preference pairs (with answers scored by the reference model), and as the reference for BLEU/ROUGE and for the case-study judgments. Under this protocol, high BLEU/ROUGE scores and low 'hallucination' rates can reflect imitation of DeepSeek's output style rather than clinically correct TCM formula generation. The paper does not report human expert evaluation on a held-out sample of the collected dataset, so the empirical superiority claims in Tables IV–VII are not independently verified. This affects the central empirical claim of the paper, not merely its presentation.","section":"§III-B.1 and §III-C.2"},{"comment":"The hallucination bound for DPO is invalid as written. The proof obtains Pθ(yl|x,c) ≤ P_ref(yl|x,c) · e^{β^{-1}∆} · (P_ref(yw|x,c)/Pθ(yw|x,c)), and then invokes the approximation Pθ(yw|x,c) ≈ P_ref(yw|x,c) to conclude Pθ(yl|x,c) ≤ P_ref(yl|x,c) · e^{−β^{-1}∆}. Substituting the approximation into the preceding expression gives Pθ(yl|x,c) ≤ P_ref(yl|x,c) · e^{β^{-1}∆}, not the claimed e^{−β^{-1}∆} bound. The sign error is load-bearing because the stated exponential decay of hallucination probability with ∆ is the entire content of Proposition 4.","section":"§III-F.4, Proposition 4"}],"minor_comments":[{"comment":"The definitions of intra-category and inter-category relationships are swapped between Section III-B.3 and Section III-B.4; the two paragraphs should be reconciled.","section":"§III-B.3 and §III-B.4"},{"comment":"Inference FLOPs are reported inconsistently: Table IV lists GraphRAG and ZhiFangDanTai as ∼1.68·10^5T, while Table IX lists the same entries as ∼1.68·10^6T; the source of the order-of-magnitude discrepancy should be clarified.","section":"Table IX vs. Table IV"},{"comment":"Reference [22] attributes Qwen2.5 to 'ByterDance'; the model is released by Alibaba, and the reference should be corrected.","section":"References"},{"comment":"There are numerous typographical errors, including 'heal th', 'T raditional', 'Base on', and 'r' in Section III-B.4; a careful proofread is needed.","section":"Throughout"},{"comment":"The learning rate is reported as 0.1, which is unusually high for LoRA fine-tuning of a 7B model; this should be double-checked against the actual training configuration.","section":"§IV-A"},{"comment":"The notation F, F_x, and F\\F_x is introduced without a clear definition of the fact set, and the relationship between ε and retrieval quality is asserted rather than defined.","section":"§III-F.3 and §III-F.4"}],"recommendation":"reject","confidential_remarks":"In my assessment the manuscript is not acceptable in its current form: the theoretical proofs contain load-bearing algebraic and logical errors, and the empirical evaluation is circular because DeepSeek-generated content is used as both training signal and evaluation reference. These problems go beyond presentation and would require re-deriving or removing the theory and re-validating the dataset with independent expert ground truth before the claims could be assessed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The empirical core is real and the theory is not. That's the short version.\n\nWhat's genuinely useful: an 80k-formula TCM knowledge graph, Leiden community detection, and a fine-grained instruction dataset with sovereign/minister/assistant/courier roles, contraindications, pulse/tongue diagnostics. The model and dataset are open-sourced, and the ablations (top-k, GPT vs no GPT, base model, DPO on/off) are honest and thorough. Tables are internally consistent and show stepwise gains from GraphRAG over RAG, and from SFT/DPO. For a low-resource domain like TCM-AI, that's a real artifact.\n\nThe soft spots are not minor. Proposition 1's proof has two unbridgeable gaps: the gradient-norm lower bound by squared mutual information is asserted with no link to the loss, and after 'integrating over T steps' the bound changes from γ²/(2β) to γ/β with the T-dependence disappearing. That's a load-bearing internal contradiction. Proposition 2's uniform preference assumption makes the claimed improvement vanish. Propositions 3/4 mix events and flip signs. The theory section is not a polish issue; it is unsupported.\n\nThe evaluation also has a closed loop: DeepSeek generates the fine-grained references used for SFT, DPO, and BLEU/ROUGE scoring, with only 50 human seeds as an anchor. Gains could reflect imitation of DeepSeek's style rather than clinical correctness. No error bars or significance tests are reported. The citation pattern is unremarkable—prior RAG/GraphRAG work is cited appropriately.\n\nWho is this for? Groups working on medical LLMs or TCM informatics will want the dataset and possibly the model. The proofs should not be relied on. As a referee, I would not desk-reject: the empirical artifact deserves expert scrutiny, and the theory errors are instructive. But I would send it back for major revision, demanding rigorous proofs or removal of the claims, and an evaluation against an independently curated gold set with human scoring and variance reporting. Current form: a strong draft, not a valid publication.","headline":"Solid GraphRAG+fine-tuning system for TCM with open-sourced data, but the headline proofs are invalid and the evaluation closes the loop with the same model that generated the references.","tokens_in":31601,"tokens_out":5575,"would_cite":false,"duration_ms":48517,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A graph-retrieval plus fine-tuning pipeline is claimed to reduce errors and hallucinations in TCM formula generation.","keywords":["Traditional Chinese Medicine formulas","GraphRAG","retrieval-augmented generation","LLM fine-tuning","direct preference optimization","hallucination reduction","knowledge graph","explainable generation"],"falsifier":"A reader could run a blinded expert review: have TCM clinicians judge whether outputs are clinically sound, and compare ZhiFangDanTai against a fine-tuned baseline using fully human-authored reference answers on the same cases. If the GraphRAG advantage disappears under expert-judged correctness, the claim that it reduces hallucination would be falsified.","tokens_in":30449,"feed_emoji":"⚕️","tokens_out":7703,"duration_ms":66978,"temperature":0.7,"pith_summary":"The paper is trying to establish a specific recipe for AI-generated Traditional Chinese Medicine formulas: build a knowledge graph from 80,000 classical formula records, cluster it into seven fine-grained topics, retrieve relevant community summaries for a patient's symptoms, and use those summaries to fine-tune a large language model in two stages (supervised fine-tuning then preference optimization). The authors claim this recipe provably lowers both generalization error and hallucination rates compared with fine-tuning alone, and that the full pipeline beats existing LLM-based TCM models on both a collected test set and real-world clinical consultations. This matters because prior systems either returned formulas without explanations, generated detailed but hallucinated explanations, or could not integrate external knowledge cleanly. A working version of the claim would give clinicians and patients a decision-support tool that outputs structured, reasoned formulas at roughly the inference cost of calling an LLM directly.","feed_headline":"Graph retrieval plus fine-tuning cuts TCM formula errors","feed_subtitle":"Combining community-based retrieval with SFT and DPO provably lowers hallucination in TCM formula generation.","key_machinery":"The load-bearing machinery is the GraphRAG retrieval pipeline coupled to two-stage fine-tuning. GraphRAG extracts entities and relations from TCM documents, builds a knowledge graph, detects hierarchical Leiden communities aligned to seven explanation facets (disease, recommended formula, herbs, symptoms and population, pulse and tongue, contraindications, preparation), and answers a symptom query by retrieving top-$k$ answers per community and summarizing them into a global context $c$. SFT trains the LLM to predict the reference answer $y$ from $(x,c)$, and DPO aligns the model toward preferred answers $y_w$ and away from rejected answers $y_l$. The carrying identity is the mutual-information gap $I(y;c|x) \\ge \\gamma$, which converts retrieval quality into a guaranteed reduction in generalization error, plus the preference-strength term $E[\\Delta]$ in the DPO bound.","core_discovery":"The paper's central discovery, stated on its own terms, is that a system which retrieves structured TCM knowledge from a graph and then fine-tunes the generator with SFT plus DPO outperforms both retrieval-only and fine-tuning-only alternatives across a battery of standard and TCM-specific metrics, on both a collected corpus and a zero-shot clinical dataset. It claims theoretical backing for this order: the generalization error upper bound becomes $E(\\theta_{\\mathrm{Final}}) \\le E(\\theta_{\\mathrm{SFT}}) - \\gamma/\\beta - E[\\Delta]/\\beta$, where $\\gamma$ lower-bounds the mutual information $I(y;c|x)$ contributed by graph retrieval and $\\Delta$ is the DPO preference strength; hallucination is bounded by $P_{\\mathrm{hall}}(y|x,c) \\le \\varepsilon + \\delta$, and DPO further suppresses rejected outputs to at most $P_{\\mathrm{ref}}(y_l|x,c)e^{-\\beta^{-1}\\Delta}$. In words, retrieval supplies information the model could not otherwise know, and preference alignment drains probability mass from hallucinated candidates. The paper interprets its experimental improvements as the empirical face of these bounds.","pith_inferences":["Editorial inference: the same recipe could transfer to other knowledge-intensive structured-output domains, such as herb–drug interaction warnings or clinical guideline summaries, whenever reference documents can be organized into a knowledge graph.","Editorial inference: because the reference texts were mostly machine-generated, a direct test of the hallucination claim should separate stylistic imitation from clinical fidelity by re-annotating a held-out set with human TCM experts.","Editorial inference: the conflict-aware instruction sets suggest a testable extension in which the model is scored on how often it emits explicit warning messages when given contradictory sources.","Editorial inference: the theoretical bounds tie hallucination to retrieval quality $\\varepsilon$, so a measurable prediction is that deliberately degrading retrieval (for example, using only one community) should raise hallucination rates approximately linearly."],"forward_implications":["GraphRAG plus fine-tuning should dominate retrieval-only and fine-tuning-only models on explainable TCM formula generation, with the largest gap on fine-grained facets such as contraindications and tongue/pulse diagnosis.","Adding preference alignment after supervised fine-tuning should further reduce the probability that the model emits a rejected or hallucinated answer, by an amount tied to the preference margin $\\Delta$.","A model trained this way should transfer zero-shot to clinical records outside its training distribution, supporting cold-start use as a decision-support assistant.","Skipping a generative-pretraining stage on the collected corpus is not only cheaper but should avoid distributional drift away from the downstream task.","The six TCM-specific evaluation metrics should show consistent gains rather than trade-offs, meaning the system improves factual support while retaining formula-compatibility and logical coherence."],"supporting_citations":[{"why":"Supplies the GraphRAG method—entity extraction, knowledge-graph construction, community detection, and map-reduce summarization—that the paper adapts to TCM.","marker":"[10]"},{"why":"The LLM used to generate the fine-grained TCM explanations from 50 human seed examples, providing the reference texts for SFT, DPO, and evaluation.","marker":"[12]"},{"why":"Defines the DPO loss used for preference alignment after SFT, the stage that yields the $\\Delta$-dependent bound on hallucinated outputs.","marker":"[15]"},{"why":"Provides the open 7-billion-parameter base model that is fine-tuned and also serves as the main plug-and-play baseline.","marker":"[7]"},{"why":"Baseline fine-tuned TCM model that motivates the need for fine-grained explanations beyond sovereign/minister/assistant/courier roles.","marker":"[17]"},{"why":"Supplies the atomic fact-support metric used to quantify hallucination in generated answers.","marker":"[37]"},{"why":"Leiden algorithm used for hierarchical community detection in the TCM knowledge graph, aligning communities with the seven explanation facets.","marker":"[25]"}],"fun_headline_variants":["Graph retrieval + fine-tuning provably reduce TCM formula errors","Less hallucination in TCM formulas via graph retrieval + SFT/DPO","Graph-augmented LLM fine-tuning beats TCM baselines on clinical data","Provably better TCM formulas via graph retrieval + DPO","Graph RAG + DPO fine-tuning shrinks TCM formula hallucinations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the detailed reference answers used for training and scoring are clinically correct, even though most were produced by an LLM from fifty human seed examples; if those references contain systematic errors or a consistent stylistic bias, the reported gains may measure imitation of that LLM rather than genuine improvement in formula quality.","fun_headline_variants_meta":{"raw":{"variants":["Graph retrieval + fine-tuning provably reduce TCM formula errors","Less hallucination in TCM formulas via graph retrieval + SFT/DPO","Graph-augmented LLM fine-tuning beats TCM baselines on clinical data","Provably better TCM formulas via graph retrieval + DPO","Graph RAG + DPO fine-tuning shrinks TCM formula hallucinations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000834,"raw_usage":{"total_tokens":3681,"prompt_tokens":1032,"completion_tokens":2649,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":2553}},"tokens_in":648,"tokens_out":2649,"duration_ms":17563,"temperature":1.0,"reasoning_tokens":2553,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:20:57.226398+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could run a blinded expert review: have TCM clinicians judge whether outputs are clinically sound, and compare ZhiFangDanTai against a fine-tuned baseline using fully human-authored reference answers on the same cases. If the GraphRAG advantage disappears under expert-judged correctness, the claim that it reduces hallucination would be falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the GraphRAG method—entity extraction, knowledge-graph construction, community detection, and map-reduce summarization—that the paper adapts to TCM."},{"cited_title":"For all retrieval-augmented methods (RAG, GraphRAG, RAG+Fine-tuning, RAG+SFT, GraphRAG+SFT, and ZhiFang- DanTai ), we consistently set top-kas 3","cited_arxiv_id":null,"evidence_quote":"The LLM used to generate the fine-grained TCM explanations from 50 human seed examples, providing the reference texts for SFT, DPO, and evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the DPO loss used for preference alignment after SFT, the stage that yields the $\\Delta$-dependent bound on hallucinated outputs."},{"cited_title":"For efficient similarity search, we utilize FAISS (Facebook AI Similarity Search 10), an embedding matching library implemented in LlamaIndex, to retrieve the top-k results","cited_arxiv_id":null,"evidence_quote":"Provides the open 7-billion-parameter base model that is fine-tuned and also serves as the main plug-and-play baseline."},{"cited_title":"The results, presented in Table VII, indicate that ZhiFangDanTai achieves the highest performance and the lowest hallucination rate","cited_arxiv_id":null,"evidence_quote":"Baseline fine-tuned TCM model that motivates the need for fine-grained explanations beyond sovereign/minister/assistant/courier roles."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Leiden algorithm used for hierarchical community detection in the TCM knowledge graph, aligning communities with the seven explanation facets."}],"review_version":2}