{"id":"a136a493-0aa9-402a-b1ab-59da8d1ebd52","arxiv_id":"2607.17902","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Fine-tuning small open-source LLMs on a new MeSH-derived benchmark (MeSH-Rel-4K) raises semantic-relation classification F1 by 34.1 points on average, reaching 91.6% for gemma-2-9b.","lead":"This paper builds a 4,000-pair benchmark from the MeSH biomedical vocabulary and tests five small open-source LLMs on deciding whether one topic is broader, narrower, the same as, or unrelated to another. Fine-tuning the models on the benchmark raised average F1 by 34.1 percentage points over chain-of-thought prompting, with the best model reaching 91.6% F1.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'other' class in MeSH-Rel-4K is constructed without a specified algorithm; if 'disjoint' only means no direct MeSH edge, mislabeled negatives can inflate the fine-tuning F1 gain.","rationale":"The reader's weakest assumption is exactly the construction of the 'other' class, and I agree that this is the most load-bearing concern. The central claim is a quantitative improvement in F1 on a benchmark; if the negative labels are unreliable, both the absolute F1 values and the reported delta can be artifacts of training on those labels. I checked the arithmetic in Tables 2-4 and it is internally consistent, and the released code/dataset is a genuine strength that makes the proposed test feasible. The absence of error bars and the broad claim about ontology construction are secondary; they do not threaten the measured result as directly as label contamination in the negative class. The verdict should remain CONDITIONAL: the paper is conditionally acceptable pending verification of the 'other' pair construction and re-evaluation on clean negatives.","tokens_in":11593,"tokens_out":7594,"duration_ms":70287,"concrete_test":"Reproduce the 'other' sampling using the released GitHub code. For every one of the 1,000 'other' pairs, compute whether the two MeSH descriptors are connected by any directed path in the January 2025 MeSH graph (transitive closure over mesh:broaderDescriptor and mesh:relatedConcept). If any non-negligible fraction (e.g., >5%) are connected, re-run the fine-tuned and CoT evaluations on the pairs that remain truly disconnected (or with corrected labels) and recompute the macro-F1 delta. A second, complementary check is to have two biomedical curators label a random sample of 100 'other' pairs and measure agreement with the released labels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing quantity is the 34.1-point macro-F1 gain from fine-tuning reported in §5.2 (Table 4 vs Table 3). This number depends on the correctness of the test-set labels, especially the 1,000 'other' pairs. §3.2 says these are 'semantically disjoint topics' but gives no algorithm. If 'disjoint' was implemented as 'no direct mesh:broaderDescriptor or mesh:relatedConcept edge', the negatives are not guaranteed to be unrelated: MeSH is a polyhierarchy, and two descriptors can be connected by a path of length >=2 (e.g., a shared ancestor or a chain of narrower relations) or via missing/unmaintained relations. Because the fine-tuned models are trained on exactly these labels, they can learn to output 'other' for the mislabeled pairs, which would inflate the 'other' F1 (94.3% average in Table 4) and the reported gain over CoT prompting, which is not trained on the label distribution. The paper provides no manual verification or transitive-closure check for the negative class. This is a correctness risk in the benchmark, not merely a framing issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MeSH-Rel-4K, a dataset of 4,000 relation pairs sampled from MeSH, labeled as broader, narrower, same-as, or other. It evaluates five small open-source LLMs (up to 9B parameters) under three adaptation strategies: standard prompting, chain-of-thought two-way prompting, and LoRA fine-tuning. The central empirical claim is that fine-tuning improves average macro-F1 by 34.1 percentage points over CoT prompting, with fine-tuned models reaching 86.2–91.6% F1 across the test set. The authors also release code and the dataset and provide confusion-matrix-based error analysis.","tokens_in":11892,"tokens_out":3814,"duration_ms":39626,"significance":"If the empirical findings hold, the paper provides a strong demonstration that resource-efficient LLMs fine-tuned on a modest dataset can perform biomedical relationship classification at a level useful for ontology curation and evolution. The main strengths are the public release of dataset/code, the held-out test set with concept-level leakage control, the systematic comparison of prompting versus fine-tuning across multiple models, and the confusion-matrix analysis. However, the benchmark's negative-class construction and the absence of statistical replicates are important correctness risks that need to be addressed before the 34.1-point gain can be fully accepted.","major_comments":[{"comment":"The construction of the 'other' class is under-specified: the paper states only that it contains '1,000 pairs of semantically disjoint topics' and gives no algorithm or verification procedure. If 'disjoint' was implemented as 'no direct mesh:broaderDescriptor / mesh:relatedConcept edge,' then pairs connected through longer paths or missing MeSH annotations would be mislabeled as unrelated. Because fine-tuned models are trained on these labels, such false negatives could inflate the 'other' F1 (average 94.3% in Table 4) and, in turn, the headline 34.1-point fine-tuning gain over CoT prompting, which is not trained on the same label distribution. Please specify the exact sampling algorithm, report whether transitive closure or expert validation was used, and quantify the label-noise risk (e.g., by analyzing how many 'other' pairs have shared ancestors or are connected at distance ≥2).","section":"Section 3.2"},{"comment":"All reported results are point estimates from what appear to be single fine-tuning runs. There are no repeated runs, confidence intervals, or significance tests. Given that LoRA fine-tuning is stochastic and the central claim is the exact magnitude of the F1 improvement, the authors should report mean ± standard deviation over at least three independent runs (or provide a statistical significance test) for the fine-tuning experiments in Table 4. This is necessary to establish that the 34.1-point gap is not an artifact of a particular seed.","section":"Section 5.2 / Table 4"},{"comment":"The LoRA hyperparameter tuning is described only as 'multiple training iterations to empirically adjust' r and alpha. For reproducibility and to rule out selection bias, the paper should report the search space, the number of configurations tried per model, the validation metric used for selection, and whether the same validation set was used across models. Table 1 lists only the final r/alpha values.","section":"Section 3.3 / Table 1"}],"minor_comments":[{"comment":"The text says average F1-scores range 'from 86.6% to 94.3%,' but Table 4 shows same-as at 86.3%. The lower bound should be 86.3%.","section":"Section 5.2"},{"comment":"The prompt templates and the 'empirical referee rules' are only described by reference to [15]. Since this paper is self-contained in other respects, include the templates and rules in an appendix or in the repository (beyond a link) so reviewers can verify the CoT setup.","section":"Section 4.1"},{"comment":"The claim that the same-as/hierarchical confusion 'correlates with dense lexical overlap' is supported only by a single example (alitretinoin/tretinoin). Quantify this correlation, e.g., by measuring token overlap or embedding similarity for misclassified versus correctly classified pairs.","section":"Section 5.3"},{"comment":"The phrase 'near-state-of-the-art models' is not supported by any comparison with state-of-the-art methods or larger LLMs. Either remove or qualify the statement.","section":"Section 5.3 / Conclusion"},{"comment":"The dataset is described as containing '4K semantic relationships,' but the 'other' class consists of unrelated pairs. Recommend calling them 'relation pairs' or 'semantic relation instances' to avoid terminological confusion.","section":"Abstract / Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim is plausible and the error analysis is useful. The decisive issue is the under-specified construction of the 'other' class in MeSH-Rel-4K; if the code already includes a proper transitive-closure or distance-based strategy, the authors can likely address this quickly, but as written it is a genuine correctness risk. The absence of repeated runs is a second concern that should be standard for a benchmark paper. I would be willing to accept after the authors clarify the negative-sampling algorithm and provide either repeated-run statistics or a clear argument that the 34.1-point gain is robust to the observed label noise."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper's core empirical claim—fine-tuning beats prompting by a large margin on relation classification over MeSH pairs—is credible and usefully quantified. The new MeSH-Rel-4K dataset is the main contribution, and the release of code and data is a plus. Five open models, three strategies, clean tables, confusion matrices: this is the kind of paper I'd want to see in a venue for applied NLP or ontology learning.\n\nWhat's genuinely new: the dataset (4K pairs from MeSH, balanced across broader/narrower/same-as/other) and the measured F1 gains, e.g., llama-3b jumping from 25.7 to 86.2 after fine-tuning. The recipe (CoT two-way, LoRA) extends the authors' earlier engineering-domain work, so this is a domain-extension with a new benchmark rather than a novel method. That's fine, but the title's 'ontology generation' overstates what is a single four-way classification step.\n\nSoft spots, in order of seriousness. First, the construction of the 'other' class: Section 3.2 says '1,000 pairs of semantically disjoint topics' with no algorithm. If 'disjoint' means only no direct MeSH edge, then pairs related via longer paths or missing annotations are mislabeled. That's a real risk in a polyhierarchy, and it could inflate the other-class F1 (94.3% average in Table 4) and the reported 34.1-point gain. The authors should specify the negative-sampling procedure and ideally do a transitive-closure check or manual verification. Second, no repeated runs or confidence intervals; macro-F1 differences of 1-3 points between models may be noise. Third, the LoRA hyperparameter tuning is described only as 'empirically adjusted' to optimal configurations—this needs detail or a released config (they say code is on GitHub, so this might be addressed). Fourth, the abstract's claim about 'construction and evolution of specialised biomedical ontologies' goes beyond the evidence, which is relation classification only.\n\nThe stress-test concern is on target. I would not call it fatal, but it's the main thing to verify. If the negative pairs are clean, the central result holds. If not, the other-class scores drop and some of the fine-tuning gain could shrink.\n\nBottom line: worth serious peer review, with a request to tighten negative-class construction and add variance estimates. I'd cite it if I worked on ontology learning; otherwise it's a useful empirical datapoint. I'd bring it to reading group as a good example of a reproducible LLM benchmark paper, with the negative-class question as discussion fodder.","headline":"Fine-tuning small LLMs gives real F1 gains on a new MeSH relation benchmark, but the construction of the 'other' class is underspecified and could inflate the headline number.","tokens_in":12338,"tokens_out":2488,"would_cite":true,"duration_ms":21944,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned small LLMs classify biomedical topic relations at 86–92% F1, up 34 points on average.","keywords":["large language models","biomedical ontologies","ontology generation","MeSH-Rel-4K","fine-tuning","chain-of-thought prompting","knowledge organization systems","semantic relation classification"],"falsifier":"Take a random sample of the 1,000 'other' pairs and have two biomedical curators judge, using the full MeSH graph plus their own expertise, whether each pair is related through any chain of broader/narrower or synonym edges. If a substantial share (say more than 10%) is connected, the reported 'other' F1 of 94.3% is an artifact of mislabeled negatives.","tokens_in":11493,"feed_emoji":"🧬","tokens_out":3458,"duration_ms":32939,"temperature":0.7,"pith_summary":"The paper tries to establish that small, open-source language models (up to 9 billion parameters) can reliably generate biomedical ontology relations when fine-tuned on a targeted dataset, despite their poor performance under prompting. On MeSH-Rel-4K, a new 4,000-pair dataset built from Medical Subject Headings, fine-tuning raises average macro-F1 by 34.1 percentage points over chain-of-thought prompting, with all five models landing between 86.2% and 91.6%. The practical point is that manual ontology curation, a known bottleneck in knowledge organization, could be delegated to cheap, locally run models. The paper also shows that prompting alone tops out at 71.6%, and that the smallest model gains the most from fine-tuning.","feed_headline":"Fine-tuned small LLMs classify biomedical topics at 86–92% F1","feed_subtitle":"Prompting alone tops out near 72%; fine-tuning pushes five small models up 34 points on MeSH-Rel-4K.","key_machinery":"The load-bearing object is MeSH-Rel-4K, a dataset of 4,000 relation pairs sampled from MeSH: 1,000 broader, 1,000 narrower, 1,000 same-as, and 1,000 'other' pairs built from semantically disjoint topics. The experiment contrasts three adaptation strategies: standard prompting, chain-of-thought two-way prompting with bidirectional consistency and empirical referee rules, and LoRA-based fine-tuning with the simple template 'Classify the relationship between A and B'. The decisive mechanism is the fine-tuning step: training on 2,800 examples with a fixed output format teaches the models the label space, whereas prompting relies on in-context reasoning.","core_discovery":"The central claim is that targeted fine-tuning exceeds the reasoning bottlenecks of smaller LLMs for classifying semantic relationships between biomedical concepts. Across five models with 3B–9B parameters evaluated on a held-out 800-pair test set from MeSH-Rel-4K, fine-tuning yields macro-F1 scores of 86.2–91.6%, while standard prompting reaches 25.7–66.9% and chain-of-thought prompting reaches 25.7–71.6%. The paper attributes the improvement to parameter-efficient fine-tuning, which re-anchors the models onto the task's label vocabulary and output format; prompting alone, including a bidirectional two-stage chain-of-thought, cannot close the gap. Error analysis traces residual confusion to","pith_inferences":["The headline 34-point gap may shrink if the 'other' labels are noisy: since 'semantically disjoint' is not formally defined, pairs connected by indirect MeSH paths could be mislabeled as unrelated, inflating F1 on that class and the overall average.","Fine-tuned models likely capture the distribution of the training sample, which is drawn from MeSH's explicit hierarchy; on novel or emerging topics not represented by such edges, prompt-based reasoning may transfer better than fine-tuning.","A testable extension would compare fine-tuning against retrieval-augmented prompting that supplies MeSH definitions; if prompted models then match fine-tuned F1, the advantage is knowledge access rather than reasoning capacity.","The observed same-as confusion suggests a lightweight lexical-overlap heuristic, applied before or after the classifier, could recover several of the remaining errors at negligible cost."],"forward_implications":["If correct, fine-tuned sub-10B models can serve as automated ontology builders, easing the manual-curation bottleneck in knowledge organization systems.","The smallest model gains 60.5 percentage points from fine-tuning, suggesting model size is less limiting than adaptation method.","The 'other' category is easiest (94.3% average F1), so fine-tuned models can filter out hallucinated relations that would introduce cycles into an ontology.","The same-as relation is hardest (86.3% average F1), identifying equivalence disambiguation as the next target for ontology-generation pipelines.","The authors propose extending the same pipeline to other academic disciplines as the direct next step."],"fun_headline_variants":["Fine-tuning adds 34 F1 points for small LLMs on biomedical ontology","Small LLMs jump to 86-92% F1 with fine-tuning on MeSH-Rel-4K","Fine-tuned small models outpace prompting in biomedical ontology","Parameter-efficient LLMs hit high F1 after fine-tuning on MeSH"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the 'other' pairs in MeSH-Rel-4K are truly unrelated: the paper describes them only as 'semantically disjoint topics' with no stated criterion, so if some are actually connected through indirect MeSH paths or missing annotations, the high F1 on 'other' and the overall 34-point gain are inflated.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuning adds 34 F1 points for small LLMs on biomedical ontology","Small LLMs jump to 86-92% F1 with fine-tuning on MeSH-Rel-4K","Fine-tuned small models outpace prompting in biomedical ontology","Parameter-efficient LLMs hit high F1 after fine-tuning on MeSH"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000162,"raw_usage":{"total_tokens":1074,"prompt_tokens":739,"completion_tokens":335,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":250}},"tokens_in":483,"tokens_out":335,"duration_ms":3669,"temperature":1.0,"reasoning_tokens":250,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T16:38:29.088349+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the 1,000 'other' pairs and have two biomedical curators judge, using the full MeSH graph plus their own expertise, whether each pair is related through any chain of broader/narrower or synonym edges. If a substantial share (say more than 10%) is connected, the reported 'other' F1 of 94.3% is an artifact of mislabeled negatives.","supporting_citations":[],"review_version":1}