{"id":"cf86a631-520e-4dc1-b029-5553c8cce641","arxiv_id":"2505.21109","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A graph of small fine-tuned Llama experts with isolated training chunks scores 3x higher exact match than a single 8B model on Cessna repair-manual QA, but with lower ROUGE-L and METEOR.","lead":"Researchers built a graph of small fine-tuned language models, each trained on one slice of an aircraft repair manual, with a router that sends each question to the right expert. The system reports three times higher exact-match answers than one larger fine-tuned model while training 1.7 times faster, suggesting low-cost AI for engineering document QA.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SLG's core EM comparison is not yet supported: no train/test split or manual QA verification is described (Section 3.1), and the claimed 3x advantage over Llama-3.1-8B contradicts Table 6 (0.12 vs 0.05 = 2.4x).","rationale":"The reader's verdict of CONDITIONAL is appropriate. The central claim—that SLG beats conventional fine-tuning by 3x on EM—requires a valid evaluation, and the paper does not yet provide one. Section 3.1's description of QA generation and use for both training and testing leaves the existence and composition of a held-out test set ambiguous. Because the orchestrator and experts are fine-tuned on the same question set (with different answers), any test question that appears in training could produce a spuriously high EM. The reported orchestrator routing accuracy of ~70% indicates the system does not fully generalize to unseen questions, which makes the large EM gap especially suspicious if no split is documented. In addition, the paper's own numbers contradict the '3x' phrasing: 0.12/0.05 = 2.4x vs the 8B model, while 0.12/0.04 = 3.0x vs the 1B model. This arithmetical slippage suggests the headline ratio should be corrected regardless. However, the qualitative finding—that a collection of small experts can outperform a larger fine-tuned model on an exact-match task—could still be true with a proper evaluation. Therefore, the paper should be revised with a clear train/test split, human validation of a sample of synthetic QA pairs, and corrected ratios; the verdict stays CONDITIONAL.","tokens_in":11904,"tokens_out":9634,"duration_ms":101540,"concrete_test":"Re-run the full pipeline from Section 4 using a 20% held-out subset of subsections for testing, with experts and orchestrator trained only on the remaining 80%; then recompute Table 6's EM values. If the SLG-vs-8B EM gap shrinks materially, the original result stems from train/test leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 explains that QA pairs were synthesized by Llama-3.3-70B-Instruct from subsection chunks and states they are 'used for model fine-tuning and testing,' but it never documents a train/test split, a held-out set of subsections, or any human verification of the generated answers. Since the same QA pairs feed both the experts and the orchestrator, the test set may overlap with training data; any such overlap would inflate EM scores independently of the SLG architecture. The orchestrator's reported routing accuracy of ~70% (Section 4.3) shows imperfect generalization, yet the EM gap is large; if test questions were memorized from training, the gap would be an artifact. Independent of this, the paper's own Table 6 gives SLG EM 0.12 vs Llama-3.1-8B EM 0.05, a 2.4x ratio, not the '3 times better' claimed in the abstract and Section 6; the 3x figure holds only against the 1B baseline (0.04). These two issues together mean the central quantitative claim is not currently substantiated by the presented evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"SLG is a graph-based system in which an orchestrator model, built on Llama-3.2-1B-Instruct and fine-tuned with LoRA, routes a user query to one of several expert models, each fine-tuned on an isolated subsection chunk of an aircraft structural repair manual. Training questions and answers are synthesized by Llama-3.3-70B-Instruct from the chunks. The paper reports that SLG achieves Exact Match 0.12 versus 0.05 for a fine-tuned Llama-3.1-8B-Instruct and 0.04 for a size-matched standalone Llama-3.2-1B-Instruct, that it fine-tunes about 1.7 times faster than the 8B model on one RTX 4090, and that the knowledge-isolation design helps resist hallucinations. The evaluation uses ROUGE-L, EM, and METEOR, with 13 hyperparameter-tuning runs summarized in Table 3 and Figure 4. The paper also reports an orchestrator routing success rate of approximately 70% and acknowledges limitations including the absence of human evaluation and the lack of conversational memory.","tokens_in":12176,"tokens_out":6495,"duration_ms":68411,"significance":"If the empirical claims were fully supported, SLG would be a practically valuable demonstration that a collection of small LoRA-tuned experts can outperform a larger standalone model on exact engineering answer retrieval at substantially lower compute cost, with potential distributed-deployment implications. The authors make a concrete, falsifiable quantitative claim, point to public code for the fine-tuning pipeline, and ground the work in a real maintenance manual, which are strengths. However, the current evaluation has unresolved data-leakage and selection-bias risks, the headline 3x figure is not internally consistent with Table 6, and the hallucination-resistance claim is supported only indirectly. The significance is therefore conditional on a substantially stronger experimental protocol.","major_comments":[{"comment":"The manuscript never defines a train/test split or a held-out set of subsections. Section 3.1 states that synthesized QA pairs are used for model fine-tuning and testing, and Section 4.1 refers to a test dataset described in Subsection 3.1, but no partition is specified. Since the same QA pairs feed the expert models and the orchestrator, the reported EM values may reflect memorization of training questions rather than generalization; this is the principal load-bearing issue for the claimed 3x EM advantage. The authors must document the split, ensure test questions and subsections are disjoint from training, and preferably add manual verification of the synthesized ground truth.","section":"Section 3.1, Section 4.1"},{"comment":"The hyperparameters are selected from 13 runs by choosing the values exhibiting the best performance, and Table 6 then reports these best results without error bars, repeated seeds, or a separate validation set. With no holdout, part of the observed EM gap can be attributed to selection bias, especially because the tuning grid is small and only EM drives the reported advantage. Please report validation-based selection, multiple runs, and variance or confidence intervals.","section":"Section 4.2, Table 3, Table 6"},{"comment":"The headline claim of a 3x EM improvement is not supported by the paper's own Table 6: SLG EM 0.12 versus Llama-3.1-8B EM 0.05 is a 2.4x ratio, not 3x; the 3x ratio holds only against the 1B baseline (0.12/0.04). The abstract and Section 6 state '3 times better' without specifying the baseline, while Section 4.3 makes the 3x claim immediately after presenting Table 6. The text and abstract must be corrected to match the data or must explicitly state the baseline.","section":"Section 4.3, Section 6, abstract"},{"comment":"EM is not by itself a hallucination metric, and Section 5 explicitly concedes that human evaluation and fact-checking could be a more exhaustive way to estimate hallucination avoidance. Since the paper's motivation is reducing hallucinations, the conclusion that SLG has the potential to better resist hallucinations needs either direct hallucination evaluation, for example adversarial or out-of-distribution queries, or a substantially softened claim.","section":"Section 4.3, Section 5"},{"comment":"The orchestrator has a reported routing success rate of only about 70%, but Table 6 reports only end-to-end metrics. Because the orchestrator is part of the system, the 0.12 EM could be depressed by routing errors; conversely, if test questions overlap training data, routing could be artificially easy. Reporting oracle-routed EM, per-expert accuracy, and the number of experts would clarify whether the advantage comes from expert specialization rather than from routing artifacts.","section":"Section 4.3"}],"minor_comments":[{"comment":"Rows 7 and 10 have identical hyperparameter values (learning rate 1e-3, LoRA rank 16, gradient accumulation 2, LoRA alpha 8); one of them appears to be a duplicate and should be removed or replaced.","section":"Table 3"},{"comment":"The dataset description omits the number of subsections, the number of experts, and the number of synthesized QA pairs per expert, which are needed to assess the system's scale and data isolation.","section":"Section 3.1"},{"comment":"Table 4 refers LoRA alpha, LoRA r, learning rate, and gradient accumulation back to Table 3; the final best configuration should be stated explicitly so that the reported results are reproducible.","section":"Section 4.2, Table 4"},{"comment":"The cross-reference to a test dataset described in Subsection 3.1 is inaccurate because Subsection 3.1 does not define any test split; please add a dedicated dataset section.","section":"Section 4.1"},{"comment":"The metric descriptions should mention the specific implementations or packages used for ROUGE-L and METEOR, since score values can differ across implementations.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The main obstacles are the missing train/test split and the selection of hyperparameters on the test data; these directly undermine the central quantitative claim. If the authors can provide a clean holdout protocol, error bars, and correct the 3x/2.4x inconsistency, the paper could become a useful applied contribution. I would not recommend acceptance in current form. The paper's scope is better matched to an applied NLP or engineering informatics venue than to a systems-focused venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know this paper before you rely on its headline: the 3x EM advantage over Llama-3.1-8B is a misread of their own Table 6 (0.12 vs 0.05 is 2.4x), and the evaluation has no stated train/test split, so the central claim isn't currently supported. That said, the underlying idea is not silly—decomposing an engineering manual into isolated subsections, fine-tuning a small Llama-3.2-1B expert per chunk, and routing queries through an orchestrator is a cheap, locally runnable alternative to one big fine-tune. The authors are honest about the orchestrator's ~70% routing accuracy and about their EM-only hallucination check.\n\nWhat's actually new is the application of knowledge-overshadowing mitigation via chunk-isolated training data, and the system is a real build with a LoRA fine-tuning pipeline and code on GitHub. That counts in its favor. The training-time comparison (3,475s vs 5,891s) is concrete and plausible.\n\nThe soft spots are the ones the reader flagged, and I'd add the arithmetic slip. Section 3.1 says the QA pairs from Llama-3.3-70B are used for fine-tuning and testing, but there is no documented split, no held-out subsections, and no human verification of the generated answers. With 13 hyperparameter runs and 'best' metrics reported without seeds, part of the gap is selection artifact. And when softer metrics like ROUGE-L and METEOR favor the 8B baseline (0.46/0.55 vs 0.41/0.50), the hallucination claim resting on EM alone is fragile. The orchestrator's 70% success also means roughly a third of questions go to the wrong expert, which explains why ROUGE-L is lower—so the current system is not yet a reliable QA tool.\n\nThe limitations section acknowledges several of these issues, which is good. I wouldn't call this a desk-reject; it deserves a serious referee, but the referee should send it back for a proper holdout evaluation, multiple seeds, released data, and human fact-checking before the headline numbers are taken at face value. For a reading group, it's a decent case study in how easy it is to overstate gains on synthetic data. I wouldn't cite the 3x claim in my own work yet.","headline":"A plausible lightweight multi-expert system whose headline 3x EM gain is undercut by a misreported ratio and an unspecified train/test split.","tokens_in":12705,"tokens_out":2198,"would_cite":false,"duration_ms":24293,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A graph of small fine-tuned experts, each trained on one isolated subsection of an aircraft repair manual and routed by an orchestrator, reports 3x the exact-match accuracy of a fine-tuned 8B model while training 1.7x faster on a single…","keywords":["Small Language Graph","multi-expert language model","LoRA fine-tuning","engineering document question answering","hallucination mitigation","knowledge overshadowing","information extraction","structural repair manual"],"falsifier":"Run SLG and the fine-tuned Llama-3.1-8B on a test set of questions written and verified by human engineers rather than synthesized by an LLM, forcing the orchestrator to route every query to the correct expert; if SLG's exact-match advantage over the 8B model does not remain near threefold, or reverses, the reported gain is an artifact of the synthetic test set or of asymmetric routing failures rather than of knowledge isolation.","tokens_in":11696,"feed_emoji":"⚙️","tokens_out":11201,"duration_ms":95487,"temperature":0.7,"pith_summary":"The paper proposes the Small Language Graph (SLG), an architecture for adapting language models to engineering documents without large compute. Instead of fine-tuning one big model on an entire manual, SLG splits the manual into subsections and fine-tunes a separate 1B-parameter expert (Llama-3.2-1B-Instruct, with LoRA) on each chunk, then uses an orchestrator to route each user query to the correct expert. On a Cessna Structural Repair Manual, the authors report that SLG reaches an Exact Match score of 0.12 versus 0.05 for a fine-tuned Llama-3.1-8B model, a threefold improvement, while total fine-tuning time is 1.7x shorter (3,475 seconds versus 5,891 seconds) and the whole system runs on one RTX 4090. The authors argue that isolating knowledge by subsection prevents 'knowledge overshadowing,' the blending of overlapping training contexts that blurs meanings, and thereby reduces hallucination in exact engineering answers. If the result holds, small and medium engineering firms could deploy accurate, locally run question-answering over their own documents without expensive cloud or cluster computing.","feed_headline":"A graph of 1B experts triples exact-match accuracy vs an 8B model","feed_subtitle":"LoRA-tuned Llama-3.2-1B experts on isolated manual sections train 1.7x faster on one RTX 4090.","key_machinery":"The central mechanism is the graph of fine-tuned experts, each a Llama-3.2-1B-Instruct model adapted with LoRA (low-rank addition of trainable adapter layers) and trained only on question-answer pairs synthesized from one subsection of the aircraft repair manual, so no expert sees overlapping knowledge from another subsection. An orchestrator node, trained to output expert names as answers, routes each incoming query to the most relevant expert. The design rationale is that eliminating data overlap removes 'knowledge overshadowing,' the blending of similar contexts that the paper identifies as a hallucination source: when two procedures share an opening but diverge later, a single model trained on both blends them, whereas an isolated expert cannot. A graph library wires the nodes and orchestrator together, and the orchestrator's measured routing accuracy is about 70 percent, meaning the reported scores are a lower bound on what perfect routing would give.","core_discovery":"The central claim is that a collection of small, narrowly fine-tuned language models connected as a graph can outperform a single larger fine-tuned model on exact engineering information retrieval. Each expert is a Llama-3.2-1B-Instruct model fine-tuned with LoRA on question-answer pairs generated from a single subsection of the source manual; the orchestrator, also a LoRA-tuned Llama-3.2-1B model, maps a user query to an expert name. On the best hyperparameter configuration, SLG scores Exact Match 0.12, ROUGE-L 0.41, METEOR 0.50, while the standalone fine-tuned Llama-3.1-8B scores Exact Match 0.05, ROUGE-L 0.46, METEOR 0.55, and the standalone Llama-3.2-1B scores Exact Match 0.04, ROUGE-L 0.43, METEOR 0.51. The reported threefold Exact Match advantage is the evidence that the multi-expert graph resists hallucination by keeping each expert's knowledge isolated; the loose-overlap metrics remain comparable because they reward similar wording rather than exact reproduction.","pith_inferences":["The synthetic test set is the paper's explicit soft spot: because both training and test QA pairs come from Llama-3.3-70B-Instruct without manual verification, a human-annotated test set could reorder the models; the paper itself concedes that human evaluation would be a more exhaustive hallucination check.","Forcing correct routing with an oracle experiment would isolate expert quality from routing quality; if SLG's Exact Match jumps well above 0.12 under oracle routing, the bottleneck is the orchestrator rather than knowledge isolation.","The knowledge-overshadowing mechanism yields a direct test: deliberately merge two overlapping subsections into one expert's training data and watch Exact Match fall while ROUGE-L stays roughly constant, which would confirm the mechanism rather than just the architecture.","The authors' distributed-inference vision is plausible only after the orchestrator improves; at 70 percent routing accuracy, a peer-to-peer fleet of device-level experts would amplify routing mistakes across the network."],"forward_implications":["If the reported metrics hold, a graph of 1B-parameter experts can replace a single fine-tuned 8B model for exact-answer engineering question answering while training 1.7 times faster and running on one consumer GPU.","The threefold Exact Match advantage with comparable ROUGE-L and METEOR indicates the benefit is specific to exact, hallucination-free reproduction of procedures, not to general fluency or wording overlap.","Because the orchestrator routes correctly only about 70 percent of the time, the paper's numbers are conservative; improving the router would raise all three metrics.","The same subsection-chunk and per-chunk expert recipe should transfer to other structured engineering documentation, since the authors note that such manuals are usually divided into logical sections.","Total SLG fine-tuning takes 3,475 seconds on a single RTX 4090, making the approach feasible for small companies that cannot afford high-end servers or cloud compute."],"supporting_citations":[{"why":"Supplies the engineering corpus: an aircraft Structural Repair Manual split into subsections, where each subsection becomes one expert's isolated training chunk.","marker":"[33]"},{"why":"Synthesizes the question-answer pairs used for both fine-tuning and evaluation, thereby defining the ground truth the metrics measure.","marker":"[34]"},{"why":"Provides the small backbone model that is LoRA-fine-tuned into every expert node and the orchestrator node.","marker":"[13]"},{"why":"The larger standalone baseline that SLG is claimed to beat threefold on Exact Match.","marker":"[36]"},{"why":"The low-rank fine-tuning method that keeps training feasible on a single 24 GB GPU and is shared across all compared models for fairness.","marker":"[24]"},{"why":"Supplies the data-overlap mechanism that motivates isolating each expert's training data to prevent hallucination.","marker":"[16]"},{"why":"The graph library cited by the paper that connects the orchestrator to the expert nodes, enabling query routing across the graph.","marker":"[35]"}],"fun_headline_variants":["Graph of 1B experts triples exact-match score vs 8B model","Small expert graph delivers 3x better exact match than 8B","Tiny experts in graph: 3x exact match, 1.7x faster training","Lightweight expert graph triples EM, cuts fine-tune time"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire comparison assumes that the question-answer pairs generated automatically by Llama-3.3-70B-Instruct are correct ground truth, that every question maps to exactly one subsection, and that subsection boundaries really isolate knowledge; if the synthetic pairs are noisy, the questions are misplaced, or the overlapping contexts still leak, the threefold exact-match gap may reflect test-set artifacts or routing errors rather than genuine hallucination resistance.","fun_headline_variants_meta":{"raw":{"variants":["Graph of 1B experts triples exact-match score vs 8B model","Small expert graph delivers 3x better exact match than 8B","Tiny experts in graph: 3x exact match, 1.7x faster training","Lightweight expert graph triples EM, cuts fine-tune time"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000238,"raw_usage":{"total_tokens":1568,"prompt_tokens":1057,"completion_tokens":511,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":673,"completion_tokens_details":{"reasoning_tokens":426}},"tokens_in":673,"tokens_out":511,"duration_ms":5645,"temperature":1.0,"reasoning_tokens":426,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:35:27.644838+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SLG and the fine-tuned Llama-3.1-8B on a test set of questions written and verified by human engineers rather than synthesized by an LLM, forcing the orchestrator to route every query to the correct expert; if SLG's exact-match advantage over the 8B model does not remain near threefold, or reverses, the reported gain is an artifact of the synthetic test set or of asymmetric routing failures rather than of knowledge isolation.","supporting_citations":[{"cited_title":"Single Engine Models 172, 182, T182, 206 AND T206 1996 And On","cited_arxiv_id":null,"evidence_quote":"Supplies the engineering corpus: an aircraft Structural Repair Manual split into subsections, where each subsection becomes one expert's isolated training chunk."},{"cited_title":"meta-llama/Llama-3.3-70B-Instruct","cited_arxiv_id":null,"evidence_quote":"Synthesizes the question-answer pairs used for both fine-tuning and evaluation, thereby defining the ground truth the metrics measure."},{"cited_title":"meta-llama/Llama-3.2-1B-Instruct","cited_arxiv_id":null,"evidence_quote":"Provides the small backbone model that is LoRA-fine-tuned into every expert node and the orchestrator node."},{"cited_title":"meta-llama/Llama-3.1-8B-Instruct","cited_arxiv_id":null,"evidence_quote":"The larger standalone baseline that SLG is claimed to beat threefold on Exact Match."},{"cited_title":"LoRA: Low-Rank Adaptation of Large Language Models","cited_arxiv_id":null,"evidence_quote":"The low-rank fine-tuning method that keeps training feasible on a single 24 GB GPU and is shared across all compared models for fairness."},{"cited_title":"LangGraph","cited_arxiv_id":null,"evidence_quote":"The graph library cited by the paper that connects the orchestrator to the expert nodes, enabling query routing across the graph."}],"review_version":1}