{"id":"275c57ba-47d2-4fd9-ae4a-96c1579d6ae8","arxiv_id":"2411.19064","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"WTS couples retrieval-augmented generation with an LLM-built, evolving domain knowledge graph and reports SOTA gains, but its main experiments use test-set gold answers to construct the graph.","lead":"This paper describes a system that lets a large language model build its own specialty knowledge graph from questions and answers, then uses that graph to retrieve facts for later questions. The authors report large accuracy gains on medical and science question sets, but the evaluation leaks the answers of earlier test questions into the knowledge base, so the gains are not trustworthy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main-table gains likely reflect test-label leakage: DKG evolution in §3.2.2 uses the gold answer α*_q to update the DKG before later test questions are answered, so the 11.3% SOTA claim is unsupported without a held-out evaluation.","rationale":"The reader's weakest assumption identifies exactly the load-bearing flaw: the DKG is evolved using gold answers from the same test set on which accuracy is measured. Section 3.2.2 makes this explicit with T+_q = LLMGen(q, α*_q, \\bar T_q^{(D)}), and Section 3.3 shows the updated DKG is used for all subsequent questions. Section 4.1.2's empty-initialization statement plus the absence of any train/evolution split for Table 1 means the reported comparisons to GPT-4o, CoT, and ToG are not evaluations of generalization to unseen questions. A legitimate apprenticeship setting could use gold answers during deployment, but then the correct baselines are systems given the same feedback budget, not static prompting baselines. The paper's own mastership split on ChatDoctor5k confirms the clean-evaluation gains are far smaller, supporting the leakage interpretation. No code or error bars are provided, but the decisive issue is the evaluation protocol. My independent read agrees with the reader's REJECT; no verdict adjustment is needed.","tokens_in":34411,"tokens_out":3861,"duration_ms":37724,"concrete_test":"Re-run the MedMCQA multiple-choice experiment with a clean split: first run WTS over a subset of the official training set (e.g., 800 randomly sampled questions with gold answers) to construct and evolve the DKG, then freeze the DKG and evaluate on the held-out test subset. Compare WTS(GPT-4o) accuracy against the GPT-4o I/O baseline using the same retrieval depth and hyperparameters reported in Table 3. If the ~11.3% relative improvement collapses to noise, the headline gain is attributable to test-label leakage; if the margin persists, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2.2 defines domain knowledge generation as T+_q = LLMGen(q, α*_q, \\bar T_q^{(D)}), explicitly using the gold answer. Section 3.3 then updates G_q to G_{q+1} with T+_q, so every later question retrieves triples produced from earlier questions' gold answers. Section 4.1.2 states 'For all datasets, WTS is initialized with an empty vector database as its DKG,' and no separate training/evolution split is described for the main Table 1 experiments; the natural reading is that WTS processes the test set sequentially, with each question's gold answer feeding the DKG used for subsequent questions. On standard benchmarks this is label leakage: the system is evaluated on the same examples that supply its knowledge. The only clean evaluation in the paper, Appendix Table 6's 800/200 apprenticeship/mastership split on ChatDoctor5k, shows a much smaller improvement (roughly 0.009 BERTScore over GPT-4o) than the headline 11.3% relative accuracy gain on MedMCQA, and it is confined to one dataset. The system design and prompts are clear enough that a proper split could be run, but as reported the central empirical claim that WTS 'surpasses the previous SOTA' is not supported. This is a benchmark-protocol problem, not an internal inconsistency of the method.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Way-to-Specialist (WTS), a retrieval-augmented generation framework that couples a DKG-augmented LLM with LLM-assisted DKG evolution. The DKG starts empty and is populated with triples generated from each answered question, and those triples are then used to answer later questions. The authors claim state-of-the-art performance on four specialized domains with up to 11.3% improvement on MedMCQA, based on experiments over six datasets spanning medical, natural science, social science, linguistics, and general QA.","tokens_in":34674,"tokens_out":6498,"duration_ms":54180,"significance":"The 'LLM-circle-arrow-KG' loop is conceptually appealing and, if validated, could reduce the need for domain-specific fine-tuning. The algorithm and prompt templates are described in enough detail to make a proper re-evaluation feasible. However, the main empirical evaluation suffers from label leakage because gold answers from the test set are used to evolve the DKG for subsequent test questions; the claimed SOTA gains are therefore not supported. The only clean split (Appendix Table 6) shows a much smaller gain on a single dataset. As such, the significance of the contribution is currently unsubstantiated.","major_comments":[{"comment":"The evaluation protocol leaks gold answers into the knowledge base. In §3.2.2, the domain knowledge generation step is defined as T+_q = LLMGen(q, α*_q, ...), where α*_q is the gold answer; §3.3 then updates the DKG to G_{q+1} using T+_q before question q+1 is answered. Since §4.1.2 states that WTS is initialized with an empty DKG for all datasets and the main experiments process the test set sequentially, the DKG used for later test questions contains triples derived from earlier test questions' gold answers. This constitutes label leakage and invalidates the headline results in Table 1, including the reported 11.3% relative improvement on MedMCQA. The only clean evaluation, Appendix Table 6, uses a separate 800/200 apprenticeship/mastership split on ChatDoctor5k and shows a BERTScore gain of 0.009 over the GPT-3.5 baseline (0.792 vs. 0.783), far smaller than the main reported gains. The main experiments must be re-run with an apprenticeship split (or a pre-built DKG) and then evaluated on held-out questions.","section":"§3.2.2, §3.3, §4.1.2"},{"comment":"The maximum retrieval depth D is selected per dataset on the test set. Table 3 reports accuracy for varying D on ChatDoctor5k, PubMedQA, and MedMCQA, and Table 5 lists the D values used to produce the main Table 1 results. Because D is chosen by comparing performance on the same data that is later reported, the main results are optimistically biased by test-set overfitting. A validation split or a single fixed D across all datasets is required to support the reported performance.","section":"§4.2.1, Tables 3 and 5"},{"comment":"Figure 7 presents the accuracy improvement over GPT-3.5 as the number of processed Q&A samples increases, along with DKG size, for the medical datasets. Since these curves are produced on the same test set whose gold answers are used to evolve the DKG, the upward trend is expected under the leakage protocol and does not provide evidence of genuine progressive specialization. The fluctuation explanation in the text ('uncertain knowledge overlap') does not address this confound.","section":"§4.2.2, Figure 7"},{"comment":"The 'previous SOTA' claim is not supported by the chosen baselines. The comparison includes only I/O prompting (GPT-3.5, GPT-4o), CoT, and ToG; no other KG-augmented RAG systems or domain-specialized LLMs (e.g., medical RAG methods such as MindMap or MedConQA) are included. Even setting aside the leakage issue, exceeding these three baselines does not establish that WTS surpasses the previous state of the art in four specialized domains.","section":"§4.1.1 and Table 1"}],"minor_comments":[{"comment":"Several figures and body text passages contain corrupted tokens beginning with '/uni00000026/...' (e.g., the axis labels in Figure 4 and the curve labels in Figure 7). These need to be repaired before publication.","section":"Figures 4–8 and body text"},{"comment":"The dataset is referred to as 'Simple Questions' in Appendix B.1 and as 'SimpleQA' in Table 1 and §4.2.1; please unify the naming.","section":"Appendix B.1 and Table 1"},{"comment":"The statement that GPT-4o underperforms GPT-3.5 on ChatDoctor5k because of GPT-4o's 'conservation' is speculative and should be supported with evidence or removed.","section":"§4.2.1"},{"comment":"Table 5 reports for MedMCQA(Multi) an execution time of 17.17s at D3 versus 18.77s at D2, which is non-monotonic; please verify the values or explain the variance.","section":"Table 5"}],"recommendation":"reject","confidential_remarks":"The central empirical claim is invalidated by gold-answer leakage in the evaluation protocol. The only clean split shows a much smaller gain on one dataset, and the hyperparameter D is tuned on the test set. In its current form the paper does not make a supported contribution; a proper re-evaluation with a held-out split and broader baselines would be needed before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the system design is genuinely new, but the headline evaluation is compromised. The DKG is evolved from the gold answers of the very test questions being scored, so the reported 'learning' is partly the model reading the answer key. I agree with the stress-test note: Section 3.2.2 defines T+_q = LLMGen(q, α*_q, ...), and Section 4.1.2 says WTS starts with an empty DKG on the test set, with questions processed sequentially. That means later test questions retrieve triples generated from earlier test answers. On standard benchmarks that's label leakage. The one clean experiment, the apprenticeship/mastership split on ChatDoctor5k in Appendix Table 6, shows a much smaller gain (BERTScore 0.792 vs 0.777 for GPT-4o). So the 11.3% SOTA claim on MedMCQA is not supportable as presented.\n\nWhat's actually new: the close-the-loop idea — RAG from an initially empty DKG that the LLM populates as it answers — is a reasonable system-level contribution relative to KAPING, ToG, and static DKG RAG. The retrieval-prune-reason pipeline and the redundancy-aware update are clearly described, and the prompts are in the appendix. The writing is honest enough to describe the apprenticeship/mastership split, which is the right instinct, and the ablation of retrieval mechanisms is useful.\n\nSoft spots beyond the leakage: retrieval depth D is chosen per dataset after inspecting performance, there are no error bars or repeated runs, and no code or data release. Those are secondary, but they compound the main problem. Also, the 'no specialized training' framing undersells the fact that the DKG is being trained on the test set.\n\nWho should read it: people working on RAG and KG-augmented LLMs will find the system description worthwhile, but they should not take the tables at face value. The idea deserves a proper test with a held-out evolution split; the current paper is a well-described prototype with an invalid evaluation.\n\nRecommendation: I'd send it to peer review rather than desk-reject, because the design is clear, the flaw is fixable, and a serious referee can ask for a clean split. But as it stands, the empirical claims should not be accepted.","headline":"Clever closed-loop RAG/KG system, but the main evaluation leaks test labels through the DKG, so the headline gains are unsupported.","tokens_in":35268,"tokens_out":2434,"would_cite":false,"duration_ms":19944,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T50","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"A knowledge graph that grows as an LLM answers questions can specialize it for a domain, with no fine-tuning.","keywords":["specialized LLMs","domain knowledge graph","retrieval-augmented generation","knowledge graph evolution","LLM-KG feedback loop","domain-specific question answering","knowledge triple generation","vector database retrieval"],"falsifier":"Run WTS with the DKG grown only from a disjoint training split, never touching test gold answers, and compare against the sequential-apprenticeship numbers: if the 11.3% margin over the prior SOTA shrinks or disappears, the per-question gold answers entering the graph carry the reported gain. A second check: if accuracy over baseline rises steadily with question position in the test sequence, the graph is genuinely accumulating useful knowledge, while a flat profile would point to prompt-side effects.","tokens_in":34155,"feed_emoji":"🔁","tokens_out":12193,"duration_ms":92353,"temperature":0.7,"pith_summary":"This paper claims that a general-purpose LLM can become a domain specialist without any fine-tuning, by building and continuously consulting its own domain knowledge graph as it answers questions. The Way-to-Specialist (WTS) framework closes a loop between two roles for the same LLM: a reasoner that retrieves and prunes knowledge triples to answer the current question, and a knowledge engineer that converts the question plus its gold answer into new triples that update the graph for the next question. Because the graph can start empty, the method targets domains where no curated knowledge graph exists. Across six QA datasets in five domains, the authors report that WTS surpasses the previous state of the art in four specialized domains, with the largest gain a relative 11.3% accuracy improvement on MedMCQA multiple-choice questions.","feed_headline":"11.3% gain: an LLM grows its own graph as it answers questions","feed_subtitle":"No fine-tuning: the loop builds a domain graph as it answers, topping static-KG and prompting baselines in four fields.","key_machinery":"The load-bearing mechanism is the LLM$\\circlearrowright$KG feedback loop: bidirectional enhancement between the LLM and a vector-database-stored domain knowledge graph (DKG), a collection of (subject, relation, object) knowledge triples. In the DKG-Augmented LLM half, entity extraction, iterative depth-wise retrieval (exact match plus cosine-similarity filtering with gap threshold $L$), LLM semantic scoring with top-$K$ pruning, and an LLM confidence-based early exit decide what knowledge reaches the prompt. In the LLM-Assisted DKG Evolution half, a schema-free LLM prompt generates candidate triples from question, gold answer, and retrieved triples, and a redundancy check (exact existence plus similarity threshold $L'$) admits only new knowledge. The connector is the evolving DKG itself: it is empty at initialization, grows as questions are answered, and is the sole channel through which experience carries over.","core_discovery":"WTS's central claim is that specialization can be achieved as a closed loop rather than as a training procedure. In the LLM$\\circlearrowright$KG paradigm, the same LLM plays both roles: as a reasoner it retrieves question-relevant triples from a domain knowledge graph (DKG), prunes them by semantic relevance, and reasons over them; as a knowledge engineer it generates new triples from the question, its retrieved context, and the gold answer, checks them for redundancy, and inserts them into the DKG. The loop closes because the DKG used to answer question $q$ is exactly the DKG that the previous questions built. The paper validates this on six QA datasets across five domains using GPT-3.5 and GPT-4o as backbones, reporting that WTS exceeds the previous best prompting-based method in all four specialized domains and reaches a maximum improvement of 11.3% over the current SOTA.","pith_inferences":["A direct test of the loop would split each test set by position: if accuracy over baseline rises as the graph grows across the first, second, and third thirds of the sequence, the accumulated graph is the carrier; a flat profile would suggest the gains come mainly from the richer prompt format rather than the stored triples.","The apprenticeship protocol feeds every test question's gold answer into the graph before the next question, so the headline margin mixes learning with evaluation; reading the loop as a deployment would require building the graph only during a separate apprenticeship phase and freezing it for scoring, as the paper does only for ChatDoctor5k and not for the datasets behind the 11.3% figure.","Because LLM-generated triples are inserted after redundancy checks but no fact verification, a wrong triple can bias later retrievals; scoring each candidate triple's confidence before insertion would reveal how much of the gain depends on graph quality rather than graph quantity.","The Mastership stage replaces gold answers with user feedback, which suggests a product path: the same loop could adapt a general assistant to a specific user's vocabulary and needs over time, with the DKG acting as a personalized, growing memory."],"forward_implications":["WTS with GPT-4o beats standard prompting, chain-of-thought, and the graph-reasoning baseline Think-on-Graph on all four specialized domains tested, with the largest margins on medical QA.","Because the framework starts from an empty knowledge graph, it can be deployed in domains that have no pre-built KG; the DKG is constructed on the fly from questions and gold answers.","A stronger base model (GPT-4o vs GPT-3.5) retrieves less deeply and builds a more compact DKG while scoring higher, suggesting the loop partially compensates for weaker parametric knowledge.","Deeper retrieval improves accuracy up to a point (MedMCQA single-choice rises from 0.358 at depth 1 to 0.622 at depth 3, then declines), after which irrelevant triples hurt performance and raise cost."],"supporting_citations":[{"why":"KAPING pioneered KG-augmented prompting with retrieved triples; WTS extends this line to evolving domain KGs.","marker":"[4]"},{"why":"Defines retrieval-augmented generation, the base paradigm WTS builds on for non-parametric knowledge infusion.","marker":"[21]"},{"why":"Think-on-Graph is the SOTA KG-reasoning baseline WTS must beat and the source of its iterative depth-wise reasoning evaluation.","marker":"[51]"},{"why":"Mindmap/CMCKG represents the static-DKG medical prompting paradigm WTS contrasts with.","marker":"[67]"},{"why":"Chain-of-thought prompting is the reasoning baseline WTS compares against across all datasets.","marker":"[64]"},{"why":"MedMCQA is the dataset carrying the reported maximum 11.3% performance improvement.","marker":"[38]"},{"why":"PubMedQA is the dataset where WTS reports its largest relative gains over I/O prompting.","marker":"[20]"},{"why":"ChatDoctor5k is the medical dialogue dataset used for the apprenticeship and mastership phase evaluation.","marker":"[23]"}],"fun_headline_variants":["LLM and graph co-evolve: no fine-tuning, 11.3% gain","Closed loop: LLM answers, then updates the domain graph","No training needed: LLM builds its own specialized graph","Bidirectional boost: LLM and evolving KG lift reasoning","LLM evolves a domain graph to beat SOTA in four fields"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that feeding each test question's gold answer into the growing knowledge graph before the next test question is answered is a fair way to measure the method, rather than a quiet form of label leakage.","fun_headline_variants_meta":{"raw":{"variants":["LLM and graph co-evolve: no fine-tuning, 11.3% gain","Closed loop: LLM answers, then updates the domain graph","No training needed: LLM builds its own specialized graph","Bidirectional boost: LLM and evolving KG lift reasoning","LLM evolves a domain graph to beat SOTA in four fields"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000524,"raw_usage":{"total_tokens":2587,"prompt_tokens":1053,"completion_tokens":1534,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":669,"completion_tokens_details":{"reasoning_tokens":1442}},"tokens_in":669,"tokens_out":1534,"duration_ms":13000,"temperature":1.0,"reasoning_tokens":1442,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:33:31.483912+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run WTS with the DKG grown only from a disjoint training split, never touching test gold answers, and compare against the sequential-apprenticeship numbers: if the 11.3% margin over the prior SOTA shrinks or disappears, the per-question gold answers entering the graph carry the reported gain. A second check: if accuracy over baseline rises steadily with question position in the test sequence, the graph is genuinely accumulating useful knowledge, while a flat profile would point to prompt-side effects.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MedMCQA is the dataset carrying the reported maximum 11.3% performance improvement."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ChatDoctor5k is the medical dialogue dataset used for the apprenticeship and mastership phase evaluation."}],"review_version":1}