{"id":"4cde7c06-119b-437b-9c99-31026cb8c12c","arxiv_id":"2412.08258","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Zero-shot LLMs, especially Claude 3 Sonnet and a fine-tuned 7B Mistral variant, classify semantic relations between engineering research topics with high F1 on the new IEEE-Rel-1K benchmark.","lead":"This paper tests 17 large language models on labeling pairs of engineering research topics as broader, narrower, same-as, or unrelated, using a new 1,000-pair benchmark built from the IEEE Thesaurus. The best model, Claude 3 Sonnet, reached an F1 score of 0.967, and the authors argue that small open models with chain-of-thought prompting can approach the performance of much larger proprietary models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Data contamination is the load-bearing threat: the paper never tests whether high F1 reflects memorized IEEE Thesaurus relations rather than zero-shot semantic reasoning.","rationale":"The reader's weakest_assumption already identifies the same core risk: high F1 may reflect memorization of the IEEE Thesaurus during pretraining rather than genuine zero-shot semantic reasoning. My reading of the full text confirms this is the most load-bearing issue. The paper's own Section 5.1 raises the possibility but provides no test, and the proposed concrete test would settle it. If the test fails, the central claim about reasoning capability and the model-parity conclusion would be substantially weakened; if it passes, the benchmark's validity is supported. I therefore do not change the reader's conditional verdict. Other weaknesses (absence of significance tests, hand-crafted two-way rules, no baselines) are real but secondary: they affect interpretation and robustness, whereas contamination affects the validity of the measured quantity itself. My agreement_with_reader is 'agree' because the primary load-bearing concern is the same one the reader highlighted first.","tokens_in":28513,"tokens_out":4450,"duration_ms":54596,"concrete_test":"Build a contamination-controlled subset of 200-300 topic pairs that are semantically parallel to IEEE-Rel-1K but whose surface forms could not have been memorized from the IEEE Thesaurus: either (a) use topics added to the IEEE Thesaurus after each model's training cutoff, or (b) replace topic names with meaning-preserving paraphrases not present in the thesaurus (e.g., 'deep neural networks' -> 'deep artificial neural nets'). Re-run sonnet, dolphin-mistral, and mixtral on this subset with the same prompts and compute F1. If F1 drops by more than ~0.1 relative to the original benchmark, or if same-as/broader accuracy falls toward chance, the reported scores are substantially driven by memorized surface forms rather than zero-shot semantic reasoning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that LLMs reason zero-shot about broader/narrower/same-as relations between research topics, with scores like Claude 3 Sonnet at 0.967 and Dolphin-Mistral-7B at 0.920 F1. But IEEE-Rel-1K is extracted directly from the IEEE Thesaurus v1.02 (July 2023), and several evaluated models have training data through late 2023 (GPT-4, Claude 3) or were instruction-tuned on large web corpora that very plausibly include IEEE-style scientific text. Section 5.1 itself concedes: 'the superior performance of proprietary models suggests they may have encountered scientific publications during pre-training' and cites Kandpal et al. on memorization, yet no contamination test is run. If models have memorized the thesaurus or its relations, the benchmark does not measure zero-shot reasoning; it measures retrieval of pretraining content. The parity claim for smaller quantized models would also be suspect if the gap between model classes reflects differential exposure rather than prompting or architecture. This is the single load-bearing assumption: without it, the headline results do not support the stated conclusions about general capability for ontology generation, especially generalization to topics and domains outside the memorized thesaurus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces IEEE-Rel-1K, a 1,000-pair gold standard sampled from the IEEE Thesaurus, and evaluates 17 LLMs (open, quantised, proprietary) on a four-way relation classification task (broader, narrower, same-as, other) under four zero-shot prompting strategies (standard one-way/two-way and CoT one-way/two-way). The central claim is that LLMs identify semantic topic relations with high accuracy in zero-shot settings—Claude 3 Sonnet reaches 0.967 F1, Dolphin-Mistral-7B 0.920 F1, and Mixtral-8x7B 0.847 F1—and that two-way CoT prompting yields the best results, enabling smaller quantised models to approach proprietary-model performance. The paper also provides an error analysis and releases code and data.","tokens_in":28727,"tokens_out":7231,"duration_ms":71949,"significance":"If the findings hold, the paper provides useful evidence that off-the-shelf LLMs can serve as a scalable first pass for ontology relation extraction in the engineering domain, and the released benchmark and codebase are valuable assets for future work. The strengths include the breadth of the model comparison (17 models, 4 strategies), the release of the gold standard and code, and the explicit error analysis. However, the headline claim of zero-shot semantic reasoning is currently undercut by the absence of a contamination analysis, by hand-crafted integration rules that may overfit the benchmark, and by the lack of statistical significance testing; the significance of the contribution is therefore conditional on these methodological gaps being closed.","major_comments":[{"comment":"The central claim that the evaluated LLMs perform zero-shot semantic reasoning is not separated from memorization of the IEEE Thesaurus. IEEE-Rel-1K is extracted directly from IEEE Thesaurus v1.02 (July 2023) and several models (e.g., GPT-4-Turbo, Claude 3) have training data through late 2023, so the benchmark may overlap with pretraining material. The paper itself acknowledges this risk in §5.1 ('the superior performance of proprietary models suggests they may have encountered scientific publications during pre-training') and cites Kandpal et al. on memorization, but no experiment quantifies it. I ask for a concrete contamination analysis: for example, evaluate the same models on (i) topics drawn from a different thesaurus (e.g., MeSH or ACM CCS) and (ii) synthetic or semantically unrelated term pairs designed to defeat surface-form heuristics. Without such evidence, the abstract's claim of 'zero-shot reasoning' and the parity claim for quantised models are not supported.","section":"§3.2; §5.1"},{"comment":"The two-way integration rules in §4.3 are hand-crafted and are applied directly to the test set without a validation split. Rules 3 and 4 use a length-based tie-breaker (len(tA) <= len(tB) selects broader; otherwise narrower) that is not justified and may implicitly encode a regularity of the benchmark rather than a general linguistic principle. The observed gains of two-way over one-way strategies (Tables 2, 4, 5) therefore depend on the specific rule design. The authors should report sensitivity to alternative integration rules (e.g., majority vote, always-broader, random tie-break, or a learned combination) and, ideally, select the rules on a held-out development set.","section":"§4.3"},{"comment":"All F1 scores are reported as point estimates without confidence intervals or significance tests. With 250 examples per class, a difference of 0.03 in class-level F1 is typically within sampling noise; with 1,000 examples, several of the model differences highlighted in §5.5 (e.g., sonnet 0.967 vs. gpt-4 0.948 in Table 1; dolphin-mistral 0.920 vs. dolphin-mistral-dpo 0.906 in Table 4) may not be statistically reliable. The paper should report bootstrap confidence intervals and pairwise significance tests (e.g., McNemar's test) for the central comparisons, and should state the number of independent runs and sampling temperature per model, none of which are currently given in §4.4.","section":"§5.5; Tables 1–4"},{"comment":"The construction of the 'other' class is weaker than the construction of the other classes. The 250 negative pairs were randomly generated and filtered only against thesaurus relations, but no human validation or inter-annotator agreement is reported for this class, even though the same-as class received manual expert selection. If some 'other' pairs are actually semantically related (or if some non-other relations are absent from the thesaurus and hence omitted from the positive classes), the measured precision and recall are biased in ways that affect all models and, in particular, the error analysis in §6. The authors should validate the negative pairs with at least two annotators and report agreement, or otherwise justify that random pairs from the IEEE Thesaurus are a safe source of negatives.","section":"§3.2"},{"comment":"The abstract and Section 7 claim that smaller quantised models 'require significantly fewer computational resources' or are 'more scalable and cost-efficient,' but the paper reports no runtime, cost, energy, or throughput measurements. Section 4.4 lists the hardware and services (V100/L4 GPUs, Bedrock, OpenAI API) but does not quantify the resources consumed by each model. Without such measurements, the parity claim is not supported. I recommend adding at least wall-clock time, cost per 1,000 pairs, or another efficiency metric for the compared models.","section":"Abstract; §4.4"}],"minor_comments":[{"comment":"There are several typos: 'Cluade 3 Haiku' should be 'Claude 3 Haiku', 'KoldbolAI' should be 'KoboldAI', and 'gtp-4' should be 'gpt-4'.","section":"§4.4; §5.4"},{"comment":"'IEEE Theasurus' is misspelled in the Introduction, and in Appendix B 'the model is sked to provide' should be 'the model is asked to provide'.","section":"§1; Appendix B"},{"comment":"The prompt templates are described as 'engineered through various refinements,' but the paper does not report which refinements were explored or whether the final prompt was selected on a development set; this matters because prompt tuning on the test set would confound the comparison. At minimum, state that the prompt was fixed before evaluation and describe any prompt variants tested.","section":"Appendices A and B"},{"comment":"The text states that 13 models improved and 4 declined overall but does not list which four declined; the reader must infer them from Table 5. Please state the four models explicitly.","section":"§5.5"},{"comment":"The exact API model identifiers or snapshot dates for the proprietary models (e.g., a specific GPT-4-Turbo version and Claude 3 Sonnet version) are not given; providing these would improve reproducibility and support future contamination assessments.","section":"§3.3.3"}],"recommendation":"major_revision","confidential_remarks":"The central risk is contamination, and the authors already gesture at it in §5.1; I would like to see a dedicated contamination/robustness section before acceptance. Also verify that the GitHub repository contains the exact prompt templates, model identifiers, and integration-rule code, since the current manuscript leaves several implementation details to the reader."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, this is a genuinely useful empirical artifact: the authors release IEEE-Rel-1K, a 1,000-pair benchmark built from the IEEE Thesaurus, and run 17 LLMs through four prompting strategies on the broader/narrower/same-as/other classification task, with full per-class precision/recall/F1 tables and a real error analysis. The results — Claude 3 Sonnet at 0.967, GPT-4 at 0.948, Dolphin-Mistral-7B at 0.920 — are plausible on their face and consistent with the broader LLM-evaluation picture. Second, the main interpretation is conditional: the authors claim zero-shot reasoning, but the benchmark is drawn directly from a thesaurus the models may well have seen in pretraining. The paper flags this itself in Section 5.1 and cites Kandpal et al., but runs no contamination test. That is the load-bearing gap, and it turns the headline claim from 'models reason about ontology relations' into 'models reproduce a thesaurus they may have memorized.' The narrower claim, that these models are very good at reproducing thesaurus-style relations, survives either reading.\n\nThe rest of the soft spots are standard for this genre, and the authors concede several. No non-LLM baselines. No error bars or significance tests on a small, balanced 250-pair-per-class sample. The 'other' class is built by random pairing filtered only against thesaurus relations, so some negatives are probably mislabeled. The two-way integration rules, particularly the length-based tie-breaker, are hand-tuned and could encode dataset regularities; they would benefit from a sensitivity analysis. None of this is fatal, but it means the parity claim for small quantised models depends on both the prompt choice and benchmark validity.\n\nWhat the paper does well, beyond the artifact: it is honest. Limitations are stated in Section 7, the discussion is appropriately cautious, and the error analysis actually looks at confusion patterns instead of just ranking models.\n\nBottom line: this is a solid benchmark paper for people building research-topic thesauri and knowledge graphs, and a useful case study in benchmark-validity pitfalls. It deserves a serious referee; the revisions that matter are adding a contamination check, non-LLM baselines, and some significance testing.","headline":"A solid, honest benchmark paper whose headline F1 scores are likely real, but whose zero-shot interpretation needs a contamination test before the strong claims hold.","tokens_in":29267,"tokens_out":2967,"would_cite":true,"duration_ms":31353,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large language models, including small quantized open models, can identify broader, narrower, and same-as relations between research topics zero-shot with high accuracy; top proprietary score is 0.967 F1, top open 7B model 0.920 F1.","keywords":["large language models","zero-shot learning","ontology generation","semantic relation classification","IEEE-Rel-1K","chain-of-thought prompting","quantized models","scholarly knowledge organization systems"],"falsifier":"Check whether performance collapses on relationship pairs that could not have appeared in pretraining data, for example topics introduced after a model's training cutoff; if it does, the high F1 comes from memorization, not zero-shot reasoning.","tokens_in":28299,"feed_emoji":"🧠","tokens_out":6703,"duration_ms":60748,"temperature":0.7,"pith_summary":"This paper tries to establish that large language models can take over one of the most labor-intensive steps of ontology building: deciding whether one research topic is broader, narrower, the same as, or unrelated to another. To test this, the authors built IEEE-Rel-1K, a 1,000-pair gold standard drawn from the IEEE Thesaurus, and ran seventeen LLMs under four zero-shot prompting strategies. Their central result is that several models classify these relations with high accuracy, led by Claude 3 Sonnet at 0.967 F1, and that a 7-billion-parameter quantized model, Dolphin-Mistral-7B, reaches 0.920 F1 with chain-of-thought prompting. The paper argues this closes much of the gap between small open models and large proprietary ones, making cheap automated ontology generation plausible.","feed_headline":"Quantized 7B model scores 0.92 F1 on research-topic relations","feed_subtitle":"On 1,000 IEEE topic pairs, prompting strategy often matters more than model size for zero-shot ontology building.","key_machinery":"The load-bearing object is IEEE-Rel-1K, a gold standard of 1,000 topic pairs sampled from the IEEE Thesaurus: 250 broader, 250 narrower, 250 validated same-as, and 250 unrelated pairs. The argument runs through four zero-shot prompting designs: standard one-way, standard two-way, chain-of-thought one-way, and chain-of-thought two-way. The two-way variants ask the model to classify the pair in both orders and then merge the answers with a fixed rule set (Section 4.3) that uses the inverse relationship between broader and narrower, the symmetry of same-as, and a tie-breaker based on the string length of the topic names. This combination of benchmark, prompting, and consistency rules is what lets the paper attribute the high F1 scores to reasoning rather than to model scale.","core_discovery":"The paper claims that zero-shot LLM classification of broader, narrower, and same-as relations between engineering research topics is not only feasible but highly accurate, and that prompt design drives much of the performance. On the IEEE-Rel-1K benchmark, the proprietary Claude 3 Sonnet reaches 0.967 average F1 with simple one-way prompting, while the open, 8-bit-quantized Dolphin-Mistral-7B reaches 0.920 F1 with a two-way chain-of-thought strategy. The authors interpret these results as evidence that modern LLMs already encode enough scientific knowledge to support automated ontology construction, and that smaller, cheaper models can be tuned to near-proprietary performance by asking them to reason through the relationship twice, in both topic orders, and combining the answers with hand-written consistency rules.","pith_inferences":["A contamination test on topic pairs created after each model's training cutoff would separate memorized thesaurus structure from genuine zero-shot reasoning; the paper does not run one.","The length-based tie-breaker in the two-way merging rules could be exploiting a regularity specific to IEEE-Thesaurus names; re-running on a benchmark where surface-form length is uncorrelated with the gold label would show whether the rule generalizes.","If these results transfer, the bottleneck for automated ontologies shifts from relation detection to synonymy detection, the category where even the best models lose the most points.","The same protocol could build analogous benchmarks from other controlled vocabularies, such as medicine or physics thesauri, to test whether the engineering finding is domain-general."],"forward_implications":["Automated or semi-automated pipelines can generate and update research-topic ontologies at a fraction of the manual cost, because the hard relation-labelling step can be delegated to zero-shot LLM inference.","For a given model, switching from standard one-way prompting to two-way chain-of-thought prompting can improve average F1 by more than 0.2 points, so prompt engineering is a first-order lever for ontology quality.","Quantized 7B open models can approach the accuracy of much larger proprietary systems, making on-premises or resource-limited ontology generation practical.","The same four-way relation scheme generalizes to SKOS-style broader/narrower/same-as structures, so the approach could be reused for other knowledge organization systems.","The best models still confuse same-as with hierarchical relations, so synonymy detection is the remaining bottleneck for fully automated ontology construction."],"supporting_citations":[{"why":"Supplies the zero-shot reasoning premise that the whole evaluation relies on.","marker":"[18]"},{"why":"Provides the chain-of-thought prompting method that drives the best open-model results.","marker":"[74]"},{"why":"Underpins the paper's own acknowledgment that models may memorize well-structured technical content, the key threat to validity.","marker":"[80]"},{"why":"Describes the OpenOrca dataset used to fine-tune Dolphin-Mistral-7B, the best quantized model.","marker":"[60]"},{"why":"Source for Claude 3 Sonnet and Haiku, the top proprietary performers.","marker":"[73]"},{"why":"Source for GPT-4, the second-best proprietary model.","marker":"[72]"},{"why":"Source for Mixtral-8x7B, the best full open model.","marker":"[58]"},{"why":"Format standard used to parse the IEEE Thesaurus PDF into the gold standard.","marker":"[56]"},{"why":"Defines broader/narrower/same-as conventions and 'use/used for' synonymy used for labeling.","marker":"[22]"}],"fun_headline_variants":["Quantized 7B model hits 0.92 F1 on topic-relation mapping","Clever prompts let small open models match big proprietary ones","Smaller LLMs rival Claude 3 on ontology relations via two-way reasoning","IEEE topic pairs: prompt engineering levels the LLM playing field","Zero-shot LLM prompt tricks match Claude 3 performance on topic mapping"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole result rests on the assumption that the models' high scores show genuine zero-shot understanding of topic relations, rather than memory of the IEEE Thesaurus from pretraining, and that the hand-written merging rules add no hidden test-set bias.","fun_headline_variants_meta":{"raw":{"variants":["Quantized 7B model hits 0.92 F1 on topic-relation mapping","Clever prompts let small open models match big proprietary ones","Smaller LLMs rival Claude 3 on ontology relations via two-way reasoning","IEEE topic pairs: prompt engineering levels the LLM playing field","Zero-shot LLM prompt tricks match Claude 3 performance on topic mapping"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002057,"raw_usage":{"total_tokens":8026,"prompt_tokens":985,"completion_tokens":7041,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":6945}},"tokens_in":601,"tokens_out":7041,"duration_ms":48611,"temperature":1.0,"reasoning_tokens":6945,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:01:08.436578+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check whether performance collapses on relationship pairs that could not have appeared in pretraining data, for example topics introduced after a model's training cutoff; if it does, the high F1 comes from memorization, not zero-shot reasoning.","supporting_citations":[{"cited_title":"doi:10.3789/ansi.niso.z39.4-2021","cited_arxiv_id":null,"evidence_quote":"Format standard used to parse the IEEE Thesaurus PDF into the gold standard."},{"cited_title":"Kandpal, E","cited_arxiv_id":null,"evidence_quote":"Underpins the paper's own acknowledgment that models may memorize well-structured technical content, the key threat to validity."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source for Claude 3 Sonnet and Haiku, the top proprietary performers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines broader/narrower/same-as conventions and 'use/used for' synonymy used for labeling."}],"review_version":1}