{"id":"3d61930c-a52a-4254-9f86-4a6ac8ff48ac","arxiv_id":"2507.21438","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A proposed dual-decoder LLM for autonomous ontology evolution is described but never implemented; only a prompt-based simulation with a 1.1B model on 120 evaluation examples is tested.","lead":"Evo-DKD claims a dual-decoder LLM framework for automatically updating ontologies, with one stream generating structured edits and another generating text justifications. The paper actually tests only a single-decoder prompt simulation on a small synthetic dataset, so the central architecture remains unvalidated.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The experiments compare prompt variants of a single fine-tuned decoder, not the proposed dual-decoder/gating architecture, so the central claim that Evo-DKD outperforms baselines is unsupported.","rationale":"The paper is written clearly and is candid that the experiments use a prompt-based simulation, but the presentation and claims do not track that caveat. The central claim requires the proposed gated dual-decoder architecture to explain the measured improvements. The evidence establishes only that a prompt asking for a triple plus an explanation outperforms prompts asking for one output type, on a model fine-tuned to emit triple-plus-explanation sequences. That is a different and weaker claim. This is not a dispute with an external consensus; it is an internal validity failure: the architecture described in Section 3 was not instantiated or tested. The reader flagged the same weakest assumption, and I agree. The additional metric problem (Section 4.2's 'LLM-Judge' uses the DistilBERT IMDb sentiment classifier) further erodes the quality comparisons, but the architecture-evidence gap is the decisive one. A definitive check requires implementing the gated dual-decoder or at least the pipeline control described above. Because the current evidence cannot support the central claim, the REJECT verdict is appropriate; my read does not change it.","tokens_in":9842,"tokens_out":6523,"duration_ms":73247,"concrete_test":"Run the Section 5.1 evaluation with three arms on the same 120 examples and the same fine-tuned TinyLlama checkpoint: (A) the Full Dual-Decoder prompt; (B) a pipeline control that generates the triple in one prompt and the explanation in a second prompt, then concatenates them before scoring; and (C) the actual Section 3.2 gated dual-decoder using the same backbone and 600-example training set. If (B) matches or beats (A) on the Section 4.2 metrics, the reported gain is output coverage, not coordinated decoding. If (C) does not beat (A), the architectural claims in the abstract and contributions are unsupported. Report per-domain means with 95% confidence intervals.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1/3.2 defines Evo-DKD as two parallel decoders plus an attention gating router (Eq. 1 for alpha_t, Eq. 2 for mixture). Section 4.1 states the implementation 'utilizes a single-decoder approach, subtly simulating dual-decoder functionalities via carefully designed chat-based prompting,' and Section 5.1 says the three modes differ only in prompting strategy, 'without changing model weights or architecture.' Thus the Section 5 experiments compare inference prompts of one TinyLlama-1.1B checkpoint, not the proposed architecture. The 'Full Dual-Decoder' mode is a prompt that asks for a triple followed by explanatory text. Its higher scores relative to Structured-only and Unstructured-only could be produced by the requested output format, by the model's fine-tuning distribution (Section 4.1 trained on joint triple+explanation sequences), or by evaluation artifacts, without any coordinated gating or dual decoding. The abstract, contributions, and conclusion nevertheless attribute the gains to 'the dual-decoder design and gating router.' This is the load-bearing gap: the evidence cannot establish the paper's central claim that Evo-DKD outperforms single-mode decoding, because the system called Evo-DKD in the experiments is not the architecture described in Section 3. The validation loop (Section 3.3) is never run; the RAG case study (Section 5.5) inserts a user-supplied triple and retrieves it. Section 6.4 candidly defers actual dual-decoder training to future work, but the headline claim is not correspondingly qualified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Evo-DKD, a framework for autonomous ontology evolution that combines a structured decoding stream (generating ontology edits) with an unstructured decoding stream (generating natural-language justifications), coordinated by an attention-based gating mechanism, followed by a validation and knowledge-base integration loop. Experiments use a TinyLlama-1.1B model fine-tuned on 600 synthetic examples, with three prompting modes intended to emulate structured-only, unstructured-only, and full dual-decoder behavior, evaluated on 120 author-curated examples across healthcare, semantic search, and cultural heritage domains. The paper reports that the Full Dual-Decoder mode outperforms the two single-mode baselines on triple extraction accuracy, explanation quality, and an 'LLM-Judge' score, and presents a qualitative RAG case study in healthcare. However, the implementation described in Sections 4.1 and 5.1 is a single decoder with different prompting strategies, not the dual-decoder architecture with gating defined in Section 3, and the claimed closed-loop validation is never executed in the experiments.","tokens_in":10164,"tokens_out":4862,"duration_ms":52835,"significance":"If the dual-decoder architecture with dynamic gating had been implemented and evaluated, the idea of coordinating structured ontology edits with textual justifications would be a relevant contribution to LLM-driven knowledge-base maintenance. The paper is transparent about the simulation in Sections 4.1 and 6.4, which is a point in its favor. Nevertheless, as it stands, the work is a prompt-ablation study on a single fine-tuned decoder; the central claim that Evo-DKD outperforms baselines is not supported because the system evaluated is not the system defined in Section 3. The evaluation also relies on a sentiment classifier mislabeled as an 'LLM-Judge', a small non-representative test set with no error bars, and a self-referential validation design. The significance of the claimed results is therefore not established.","major_comments":[{"comment":"The central claim is unsupported because the experimental system is not the architecture described in the paper. Section 3.1–3.2 defines Evo-DKD as two parallel decoders coordinated by an attention-based gating router (Eq. 1 for alpha_t and Eq. 2 for the mixture). Section 4.1 states: 'Our implementation utilizes a single-decoder approach, subtly simulating dual-decoder functionalities via carefully designed chat-based prompting.' Section 5.1 confirms that the three modes 'varied only the prompting strategy ... without changing model weights or architecture.' Thus the experiments compare three prompting strategies applied to one TinyLlama-1.1B checkpoint, not the proposed dual-decoder/gating system, so the abstract's and conclusion's attribution of gains to 'the dual-decoder design and gating router' is not justified.","section":"§3.1–3.2, §4.1, §5.1"},{"comment":"The claimed closed-loop validation is neither independently designed nor actually exercised. Section 3.3.1 states that the validation uses 'the LLM itself to evaluate whether the explanation indeed provides evidence for the structured claim,' which is a self-referential check. Moreover, the experiments in Section 5 never run this validation module; the Section 5.5 case study inserts the triple (Ozempic, manages, weight) directly into the knowledge graph without any demonstrated consistency or justification check. The 'closed-loop autonomous updating' behavior is therefore not demonstrated beyond a single manual insertion.","section":"§3.3.1, §5.5"},{"comment":"The 'LLM-Judge Score' is not produced by an LLM. Section 4.2 says: 'We used a DistilBERT classifier (lvwerra/distilbert-imdb) to produce the LLM-Judge Score.' This is a sentiment-analysis model, not an LLM judge. Using a sentiment classifier to measure 'the overall credibility and quality of the generated explanations' is not a valid qualitative metric, and the statements in Section 5.2.3 about an 'independent qualitative metric' and an 'external evaluator's perspective' are unsupported.","section":"§4.2, §5.2.3"},{"comment":"The quantitative evaluation is too thin to establish 'consistent outperformance.' Section 5.1 reports results on 40 examples per domain (120 total) with no error bars, confidence intervals, or significance tests. Section 4.3 reports high scores on 'test data' but does not describe any held-out split, and it is unclear how these numbers relate to the 120 curated examples of Section 5.1. The absence of any statistical analysis makes the reported improvements across modes unreliable.","section":"§4.3, §5.1"}],"minor_comments":[{"comment":"A typo appears in Section 1: 'ontologies must be continuously update' should be 'ontologies must be continuously updated.'","section":"§1"},{"comment":"The heading 'Explanation Quality (BERTScore and BLEU:)' contains a stray colon after 'BLEU'.","section":"§5.2.2"},{"comment":"The figures (Figures 2–8) are referenced but not included in the text; the manuscript should provide the actual numeric scores in tables so that the claims can be independently verified.","section":"§5"},{"comment":"Section 4.1 describes 600 synthetic training examples, while Section 5.1 describes 120 curated evaluation pairs; the relationship between these sets and the construction of the evaluation pairs should be clarified.","section":"§4.1, §5.1"},{"comment":"The RAG case study uses a Gemini-powered LLM and a semantic similarity function, but no details are given for the retrieval function, the encoding of the KG context, or the prompt used with Gemini, making the case study non-reproducible.","section":"§5.5"},{"comment":"The abstract and conclusion state that results 'confirm the contributions of the dual-decoder design and gating router,' which contradicts the explicit admission in Sections 4.1 and 6.4 that the actual dual-decoder training is deferred to future work; these statements should be reconciled.","section":"Abstract, §7"}],"recommendation":"reject","confidential_remarks":"The manuscript is candid about the simulation in Sections 4.1 and 6.4, but the abstract, contributions, and conclusion systematically overstate what has been demonstrated. The main gap is that the evaluated system is a prompt-based simulation of a single decoder, not the dual-decoder architecture with gating, so the central claim cannot be supported. The 'LLM-Judge' metric is a sentiment classifier, and the evaluation set is small and without statistical rigor. Repairing these issues would require a complete reimplementation of the architecture and a new evaluation, which is beyond the scope of a revision. The paper's fit with the journal's scope may also be questioned, as the empirical content is a prompt-ablation study rather than an architecture evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper describes a dual-decoder architecture with gating, but every experimental result comes from a single TinyLlama decoder run under three different prompts. The abstract and conclusion attribute gains to the gating router and dual decoders; that attribution isn't supported by the experiment, and the paper's own Sections 4.1 and 6.4 say the real architecture is future work.\n\nWhat's genuinely here: a clearly written proposal, an honest limitations section, and a sensible intuition that pairing a triple with a natural-language explanation can improve ontology-editing quality. The related-work placement is fine. The authors also had the decency to state, in the body, that they simulated dual decoding via prompting.\n\nSoft spots, in proportion:\n- Load-bearing mismatch: Evo-DKD as defined in Section 3 (two decoders, attention gating, Eqs. 1-2) is never built. Section 5 varies prompts only. So 'Full Dual-Decoder' is just one prompt that asks for a triple then an explanation. The gains could come from the output format, the fine-tuning distribution (they trained on joint triple+explanation sequences), or evaluation artifacts. No evidence supports the architecture.\n- Metrics: 'LLM-Judge Score' is a DistilBERT sentiment classifier (lvwerra/distilbert-imdb). That measures IMDb sentiment, not credibility or quality of explanations. Using it as a quality metric is misleading.\n- Evaluation: 120 author-curated pairs, no error bars, no baseline from prior work (e.g., LLMs4OL or OntoGenix). Precision/Recall in Section 4.3 is reported on training data and means little.\n- Validation loop in Section 3.3 is never run. The RAG case study (Section 5.5) inserts a user-supplied triple into the KG, so the downstream improvement is tautological.\n- The self-referential validation (same LLM checks its own explanation) is a limitation, and the paper acknowledges it only partially.\n\nWho is this for? Someone who wants a low-cost, prompt-based exploration of combining structured and unstructured generation for KG updates. That is a reasonable thing to study, but the paper does not deliver the architecture it announces, and the metric misuse means the empirical claims don't stand.\n\nRecommendation: I would desk-reject this. The central claim is unsupported and the evaluation is below the bar for a serious referee. The authors should either reframe the work as exactly what it is—a prompt-ablation on a fine-tuned 1B model—and drop claims about dual decoders, or actually train the gated dual-decoder and run the validation loop.","headline":"Not a dual-decoder paper—a prompt-ablation of a single 1.1B model, with a central claim the experiments cannot support.","tokens_in":10687,"tokens_out":3379,"would_cite":false,"duration_ms":34969,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that pairing structured ontology edits with natural-language justification improves autonomous LLM-driven ontology evolution more than either output alone.","keywords":["Ontology evolution","Knowledge graph maintenance","Dual-decoder architecture","Neuro-symbolic reasoning","Large language models","Gating mechanism","Closed-loop learning","Structured knowledge extraction"],"falsifier":"A decisive check would be to train or obtain a true dual-decoder model with the described attention gating, and compare it to the prompt-based single-decoder simulation on the same 120 evaluation examples. If the real dual-decoder fails to beat the simulation, or if the simulation matches a plain single-decoder given the same combined prompt template, then the observed gains come from the output format and not from dual-stream coordination. A cheaper falsifier is to run the full dual-decoder mode with the explanation placed before the triple and see whether precision drops.","tokens_in":9607,"feed_emoji":"🧠","tokens_out":5187,"duration_ms":61858,"temperature":0.7,"pith_summary":"The paper claims that autonomous ontology evolution works best when an LLM is made to produce two outputs at once: a structured ontology edit, such as a triple \"(Ozempic, manages, weight)\", and a natural-language explanation that justifies it. The proposed Evo-DKD framework pairs these outputs through an attention-based gating mechanism and a validation loop, so that only justified edits enter the knowledge base. This matters because ontologies and knowledge graphs require continuous updates, and the paper argues that pairing formal structure with textual reasoning reduces the hallucination and format problems that plague purely text-based or purely triple-based LLM pipelines. The experiments, however, evaluate a prompt-based simulation of the dual-decoder design on a single fine-tuned 1.1B-parameter model, comparing structured-only, unstructured-only, and combined modes. On those comparisons, the combined mode achieves higher triple-extraction accuracy, explanation quality, and judged reliability, which the paper reads as evidence for the dual-decoder idea.","feed_headline":"LLM ontology edits improve when paired with plain-text justification","feed_subtitle":"Combined triple-plus-explanation decoding beats structured-only and text-only baselines on precision and downstream retrieval.","key_machinery":"The central object is the dual-knowledge decoding loop. In the proposed architecture, a structured decoder generates ontology-compatible triples while an unstructured decoder generates free-text reasoning; an attention-based gating module computes a weight $\\alpha_t$ at each step to blend or switch between the two output distributions. A validation module then checks the proposed edit against ontology constraints and asks whether the textual explanation actually justifies it, and only passing edits are written back into the knowledge base, whose updated state becomes context for the next iteration. In the implemented experiments this machinery is simulated by prompting a single fine-tuned decoder in three modes—structured-only, unstructured-only, and full dual-decoder—with the same model weights in all three.","core_discovery":"On its own terms, the paper establishes that jointly generating a structured triple and a textual justification, with the explanation used to cross-check the edit before injection into the knowledge base, produces more precise and more explainable ontology updates than either stream alone. A worked healthcare case shows a new input about Ozempic being converted into the triple (Ozempic, manages, weight) with a supporting sentence, after which a retrieval-augmented QA pipeline can answer a question it previously could not answer. The paper claims this closed loop—propose, validate, inject, feed back into context—is the mechanism that makes LLM-driven knowledge base maintenance sustainable. It does not claim to have trained a real dual-decoder system; it states that the dual-decoder dynamics are approximated with carefully designed prompting on a single fine-tuned model.","pith_inferences":["Because the evaluation compares prompting modes on one model rather than a trained dual-decoder, an equally consistent reading is that simply requesting both a triple and an explanation in one prompt improves output; the architectural gating story remains untested. This is an editorial inference, not a claim the paper makes.","A testable extension would be to train an actual structured decoder and text decoder with the proposed gating and compare against the prompt-simulated mode on identical data; the paper leaves this as future work.","The gains may depend on the explanation appearing after the triple in the output; swapping the order, varying explanation length, and using a different base model would clarify whether the mechanism is coordination or merely extra inference.","The same joint-decoding idea could transfer to other schema-constrained tasks, such as database ETL or metadata extraction, wherever each structured write needs a human-auditable reason."],"forward_implications":["If the joint-generation claim is correct, knowledge graphs and ontologies can be extended from raw text with less human curation while keeping each addition auditable.","Closed-loop injection means downstream tasks that retrieve from the knowledge base inherit newly learned facts, as illustrated by the Ozempic QA example.","The validation step filters edits that fail consistency checks or lack textual support, which should reduce harmful hallucinations propagating into the ontology.","The balanced precision-recall profile the paper reports suggests the combined mode is usable in settings where both avoiding false edits and catching true ones matter.","The framework positions the LLM as an active editor of its own knowledge store, opening a path toward continual-learning systems rather than fixed models with static retrieval."],"supporting_citations":[{"why":"Provides the zero-shot LLM ontology-learning baseline that Evo-DKD claims to go beyond with internal validation.","marker":"[1]"},{"why":"Supplies the never-ending-learning paradigm that motivates Evo-DKD's closed loop, with handcrafted modules rather than neural generation.","marker":"[2]"},{"why":"Toolformer is an example of LLM tool use treating external knowledge as static, the contrast for Evo-DKD's active editing.","marker":"[4]"},{"why":"OntoGenix is the human-review post-generation baseline, contrasting with Evo-DKD's autonomous in-loop validation.","marker":"[6]"},{"why":"LLM-based knowledge graph completion is the text-to-triple conversion approach that Evo-DKD extends by adding explanations.","marker":"[7]"},{"why":"ReAct grounds the reasoning-and-acting paradigm against which Evo-DKD positions its static-consumer view of external knowledge.","marker":"[8]"},{"why":"The ontology evolution survey defines the manual and semi-automated curation problem that Evo-DKD targets.","marker":"[9]"}],"fun_headline_variants":["Dual decoding streams refine ontologies with text justification","LLM ontology edits get precision boost from text explanations","Closed-loop dual-stream LLM sharpens ontology evolution","Structured triples plus text explanations improve ontology edits","Dual-knowledge decoding yields more precise ontology updates"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that prompting a single model to output a triple followed by an explanation is a faithful stand-in for the proposed gated dual-decoder architecture, so the measured improvements are attributed to the architecture rather than to the prompt format. If that premise gives way, the central architectural claims are unsupported.","fun_headline_variants_meta":{"raw":{"variants":["Dual decoding streams refine ontologies with text justification","LLM ontology edits get precision boost from text explanations","Closed-loop dual-stream LLM sharpens ontology evolution","Structured triples plus text explanations improve ontology edits","Dual-knowledge decoding yields more precise ontology updates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000804,"raw_usage":{"total_tokens":3552,"prompt_tokens":987,"completion_tokens":2565,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":2489}},"tokens_in":603,"tokens_out":2565,"duration_ms":22612,"temperature":1.0,"reasoning_tokens":2489,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T12:45:39.156648+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check would be to train or obtain a true dual-decoder model with the described attention gating, and compare it to the prompt-based single-decoder simulation on the same 120 evaluation examples. If the real dual-decoder fails to beat the simulation, or if the simulation matches a plain single-decoder given the same combined prompt template, then the observed gains come from the output format and not from dual-stream coordination. A cheaper falsifier is to run the full dual-decoder mode with the explanation placed before the triple and see whether precision drops.","supporting_citations":[{"cited_title":"Hr- uschka, and Tom M","cited_arxiv_id":null,"evidence_quote":"Supplies the never-ending-learning paradigm that motivates Evo-DKD's closed loop, with handcrafted modules rather than neural generation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Toolformer is an example of LLM tool use treating external knowledge as static, the contrast for Evo-DKD's active editing."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ReAct grounds the reasoning-and-acting paradigm against which Evo-DKD positions its static-consumer view of external knowledge."}],"review_version":1}