REVIEW 3 major objections 7 minor 12 references
Inferring Adjective Hypernyms with Language Models to Increase the Connectivity of Open English Wordnet
T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Fine-tuning on a new 302-pair gold standard improves language models' adjective hypernym prediction, extending a wordnet-link method to adjectives.
desk verdict A genuinely new English adjective hypernymy dataset and a thoughtful theoretical framing, but the headline fine-tuning gains rest on an unclear scoring rule for multi-output models in Table 4. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the substitution-based inclusion test for adjective hypernymy, paired with sense disambiguation through Open English Wordnet definitions. A candidate hypernym is accepted when it can replace the hyponym in a sentence while preserving meaning at a general level and without contradiction; the definition of each adjective pins down the intended sense. Training turns this criterion into data: hyponym-hypernym pairs are fed to the models in a chat format with and without definitions, and the 'multiple' version of the dataset expands each hypernym to its synset synonyms so the model learns one-to-many valid answers.
What would settle it
Have a fresh set of annotators re-judge the 302 pairs from scratch in English, or test the fine-tuned models on an independently written English adjective-hypernym dataset not derived from Dutch or Polish wordnets; if agreement on the gold standard collapses or the models' advantage over zero-shot disappears, the reported gains are artifacts of the annotation and translation process.
Extended reading notes
Core claim
The central claim is that the TaxoLLaMa hypernym-discovery methodology, previously applied to nouns and verbs, transfers to adjectives when supported by a purpose-built gold standard. On the paper's own terms, adjective hypernymy is real but requires sense-level grounding: a hypernym is an adjective whose meaning includes the hyponym's meaning and that can be substituted for the hyponym in context without contradiction. The paper's 302-pair dataset operationalizes this through Open English Wordnet definitions and synset membership, and fine-tuning on it, particularly the version enriched with hypernym synonyms, raises adjective hypernym prediction from an MRR of 9.4 to 23.6 without definitions and from 25.8 to 33.3 with definitions for TaxoLLaMa, while also shifting the predicted part of speech from mostly nouns to almost always adjectives.
Load-bearing premise
The dataset built by translating Dutch and Polish wordnet pairs and manually validating them in English accurately represents English adjective hypernymy, despite annotators agreeing only moderately (Cohen's kappa about 0.61-0.65).
Editorial extensions
If this is right
- Fine-tuning on the 302-pair dataset lifts TaxoLLaMa's adjective-hypernym MRR from 9.4 to 23.6 without definitions and from 25.8 to 33.3 with definitions, and makes its outputs adjectives 95-100% of the time.
- Training with multiple synonymous hypernyms consistently beats single-hypernym training, indicating that adjective hypernymy is naturally one-to-many.
- Providing the hyponym's definition improves hypernym prediction in most settings, confirming word-sense disambiguation as a core ingredient.
- The same fine-tuning recipe works for a much smaller 360M-parameter model, suggesting the method is accessible without large compute.
- The released gold standard gives the community a first benchmark for adjective hypernymy and a basis for adding hypernym links to unlinked adjective synsets in the Open English Wordnet.
Reading between the lines
- The small size and translation-based origin of the dataset mean the reported gains may partly reflect learning the annotation style rather than general adjective semantics; an independent English-only test set would separate the two.
- The substitution criterion could be turned into a direct model probe: ask an entailment model whether replacing the hyponym with the hypernym in a sentence preserves truth, and compare those judgements with the annotated pairs.
- Because best F1 on the multiple-answer evaluation reaches only 0.26 for TaxoLLaMa-ft-multi with definitions, using the model to propose new links would need human review; the immediate practical gain is the benchmark and the part-of-speech correction, not turnkey link completion.
- The findings suggest adjective hypernymy may be better represented as a graded, context-dependent scale than as a strict noun-style hierarchy, which would align with how the dataset's 'maybe' disagreements cluster.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper addresses the lack of adjective hypernymy relations in the Open English Wordnet (OEWN). It proposes a substitution-based theoretical definition of adjective hypernymy, constructs a gold-standard dataset of 302 English adjective hypernym pairs by translating and validating pairs from Polish and Dutch wordnets, and fine-tunes two language models (TaxoLLaMa and SmolLM-360M-Instruct) on two versions of this dataset (a 'single' version with one exact hypernym and a 'multiple' version augmented with hypernym synonyms). The authors report improvements in part-of-speech prediction and hypernym prediction after fine-tuning, concluding that the TaxoLLaMa methodology can be adapted to adjective hypernymy. The paper also releases the dataset in RDF and the fine-tuned models on HuggingFace.
Significance. If the results hold, the paper contributes a novel, openly released gold-standard dataset for English adjective hypernymy, a relation absent from Princeton WordNet and OEWN, and demonstrates that large language models can be adapted to this task, with potential to extend hypernymy structure to the adjective portion of OEWN. The theoretical discussion distinguishing adjective hypernymy from noun/verb hypernymy is a useful starting point. The paper also makes its code and models publicly available, which supports reproducibility. However, the evaluation is preliminary: it relies on a small test set (91 pairs), lacks significance testing, and the scoring protocol for multi-output models is underspecified, so the reported improvements must be interpreted with caution.
major comments (3)
- [Section 7, Table 4] Table 4's scoring rule for multi-output models is not specified. The caption states that 'apart from TaxoLLaMa-Zero-shot all the other models output a single hypernym', but this is contradicted by Table 5, where TaxoLLaMa-ft-multi has precision 0.15 and recall 0.20, indicating multiple outputs. Moreover, the text in Section 7 reports that evaluating only the first-ranked hypernym from base TaxoLLaMa's output list drops its correct predictions from 14 to 7, which implies Table 4 uses an any-match convention. If any output item is counted as a hit, TaxoLLaMa-ft-multi is advantaged by generating a longer list; if only the first item is counted, the single reported value for that row is misleading. Because the headline claim that TaxoLLaMa-ft-multi achieves the best exact-hypernym scores (0.35/0.44 vs 0.32/0.31) rests on this unstated convention, please specify the exact evaluation protocol (e.g., top-1, any-match, or set-based) and report results under both conventions for all model variants.
- [Section 7, Tables 4-5] The evaluation is based on a test set of only 91 pairs, yet no confidence intervals, error bars, or significance tests are reported. For instance, the difference between TaxoLLaMa-ft-multi and TaxoLLaMa-ft-single in the no-definition column of Table 4 is 0.35 vs 0.32, which is well within the range of random variation for a sample of this size. The central claim that fine-tuning on the synonym-augmented 'multiple' dataset improves exact-hypernym prediction would be substantially strengthened by a bootstrap or exact binomial test, or at least by reporting the number of correct predictions per condition. Without such analysis, the observed improvements (and the apparent lack of improvement from definitions for TaxoLLaMa-ft-single) cannot be distinguished from noise.
- [Section 5, Dataset creation] The gold-standard dataset is constructed by translating hypernym pairs from Polish and Dutch wordnets and validating them with annotators, with Cohen's kappa of 0.61-0.65 across samples and a Fleiss kappa of 0.48 for the three-annotator subsample. The paper does not report how the 'yes-maybe' cases that were retained after discussion were resolved, nor does it provide any external validation of the translated pairs against an independent English resource (e.g., dictionary definitions or a separate substitution test). Since this dataset is the basis for both fine-tuning and evaluation, a more detailed reliability analysis (e.g., per-source agreement, examples of pairs discarded at each stage, and the resolution protocol for disagreements) is needed to support the claim that the resource is a reliable gold standard for English adjective hypernymy.
minor comments (7)
- [Section 7] The MRR values reported in the text (9.4, 25.8, 23.6, 33.3) are not shown in any table, and the scale is inconsistent with the earlier mention of 54.39 (likely a percentage). Please either report MRR in a table or remove it from the text for consistency.
- [Table 4 caption] The phrase 'before the fine-tuning' is ambiguous; it should read 'zero-shot' for the corresponding rows.
- [Section 5] Use 'Cohen's kappa' (with an apostrophe) consistently instead of 'Cohen’s kappa'.
- [Section 6.1] 'Unsloth method' should be 'Unsloth library' or 'using Unsloth'.
- [Table 5 caption] The model name 'TaxoLLama' should be 'TaxoLLaMa'.
- [Section 8] The future-work list (a-d) is inline and hard to read; consider formatting it as a proper list.
- [References] The references McCrae et al. (2019a) and (2019b) appear to cite the same paper; please verify and merge if duplicate.
Circularity Check
No significant circularity: the fine-tuning results are evaluated on a held-out split of an independently constructed dataset.
full rationale
The paper's central claim is empirical: fine-tuning TaxoLLaMa and SmolLM on a new gold-standard adjective-hypernymy dataset improves held-out hypernym prediction. The dataset is built from Dutch and Polish wordnets, translated, and human-validated, not derived from the models' outputs or from the evaluation metric. The held-out 30% test split means the reported improvements are generalization results, not fits renamed as predictions. The theoretical definition of adjective hypernymy (Section 4) is used to guide annotation, and the resulting dataset is then used as the benchmark; this is standard benchmark practice rather than a circular reduction, since the claim is that models can learn the relation as operationalized, not that the operationalization is derived from the models. Self-citations to OEWN (McCrae et al.) are resource citations, not load-bearing uniqueness or justification arguments. The Table 4 note claiming all models except zero-shot output a single hypernym is inconsistent with Table 5's P/R split for ft-multi, but that is an evaluation-convention ambiguity (a correctness risk), not a circularity of the kind where the prediction reduces by construction to the input. Overall, no self-definitional, fitted-input, or self-citation-chain circularity is exhibited.
Assumptions & free parameters
free parameters (3)
- learning rate =
2e-4
- optimization steps =
60
- effective batch size =
8 (per-device 2, accumulation 4)
assumptions (4)
- domain assumption Adjective hypernymy exists and can be defined via contextual substitution and semantic inclusion.
- domain assumption Hypernymy relations transfer from Dutch and Polish to English through translation.
- domain assumption OEWN synset membership provides reliable sense disambiguation.
- domain assumption A gold standard with moderate inter-annotator agreement (kappa 0.61-0.65) is sufficiently reliable for training and evaluating models.
Cite this review
Pith. "Pith review of Inferring Adjective Hypernyms with Language Models to Increase the Connectivity of Open English Wordnet." pith.science (2026). https://pith.science/paper/AKZLH5CM
@misc{pith2026250610715,
author = {Pith},
title = {Pith review of: Inferring Adjective Hypernyms with Language Models to Increase the Connectivity of Open English Wordnet},
year = {2026},
howpublished = {\url{https://pith.science/paper/AKZLH5CM}},
note = {Machine review of arXiv:2506.10715}
}
read the original abstract
Open English Wordnet is a key resource published in OntoLex-lemon as part of the linguistic linked open data cloud. There are, however, many links missing in the resource, and in this paper, we look at how we can establish hypernymy between adjectives. We present a theoretical discussion of the hypernymy relation and how it differs for adjectives in contrast to nouns and verbs. We develop a new resource for adjective hypernymy and fine-tune large language models to predict adjective hypernymy, showing that the methodology of TaxoLLaMa can be adapted to this task.
Reference graph
Works this paper leans on
-
[6]
plWordNet 3.0 – a comprehensive lexical-semantic resource. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2259–2268, Osaka, Japan. The COL- ING 2016 Organizing Committee. Diana McCarthy and Roberto Navigli
work page 2016
-
[1995]
WordNet: a lexical database for English.Commun. ACM, 38(11):39–41. Viktor Moskvoretskii, Ekaterina Neminova, Alina Lobanova, Alexander Panchenko, and Irina Nik- ishina. 2024a. TaxoLLaMA: WordNet-based model for solving multiple lexical semantic tasks. InPro- ceedings of the 62nd Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: ...
work page 2024
-
[2007]
SemEval- 2007 task 10: English lexical substitution task. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), pages 48–53, Prague, Czech Republic. Association for Computa- tional Linguistics. John P. McCrae
work page 2007
-
[2011]
How we BLESSed distributional semantic evaluation. InPro- ceedings of the GEMS 2011 Workshop on GEomet- rical Models of Natural Language Semantics, pages 1–10, Edinburgh, UK. Association for Computational Linguistics. Georgeta Bordea, Els Lefever, and Paul Buitelaar
work page 2011
-
[2014]
Modelling the semantics of adjectives in the ontology-lexicon interface. InPro- ceedings of the 4th Workshop on Cognitive Aspects of the Lexicon (CogALex), pages 198–209, Dublin, Ireland. Association for Computational Linguistics and Dublin City University. John P. McCrae, Alexandre Rademaker, Francis Bond, Ewa Rudnicka, and Christiane Fellbaum. 2019a. En...
work page 2019
-
[2016]
SemEval-2016 task 13: Taxonomy extraction evalua- tion (TExEval-2). InProceedings of the 10th Interna- tional Workshop on Semantic Evaluation (SemEval- 2016), pages 1081–1091, San Diego, California. As- sociation for Computational Linguistics. Jose Camacho-Collados, Claudio Delli Bovi, Luis Espinosa-Anke, Sergio Oramas, Tommaso Pasini, Enrico Santus, Vere...
work page 2016
-
[2017]
InPro- ceedings of eLex 2017, pages 587–597
The ontolex- lemon model: development and applications. InPro- ceedings of eLex 2017, pages 587–597. John P. McCrae, Christian Chiarcos, Francis Bond, Philipp Cimiano, Thierry Declerck, Gerard de Melo, Jorge Gracia, Sebastian Hellmann, Bettina Klimek, Steven Moran, Petya Osenova, Antonio Pareja-Lora, and Jonathan Pool
work page 2017
-
[2018]
SemEval-2018 task 9: Hy- pernym discovery. InProceedings of the 12th Inter- national Workshop on Semantic Evaluation, pages 712–724, New Orleans, Louisiana. Association for Computational Linguistics. R. M. W. Dixon. 1982.Where have All the Adjectives Gone?De Gruyter Mouton, Berlin, New York. Christiane Fellbaum. 1998.WordNet: An Electronic Lexical Databas...
work page 2018
Show all 12 references
-
[2021]
InProceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 9044–9062, Online and Punta Cana, Dominican Republic
Distilling relation embeddings from pretrained language models. InProceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 9044–9062, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. Ivan Vuli´c, Danie...
2021
-
[2022]
InInternational Conference on Learning Representations
LoRA: Low-rank adaptation of large language models. InInternational Conference on Learning Representations. Franz Hundsnurscher and Jochen Splett. 1982.Grundle- gung Einer Semantischen Beschreibung der Adjektive des Deutschen, pages 16–47. VS Verlag für Sozial- wissenschaften,...
1982
-
[2023]
John Lyons
Adjective scale probe: Can language models encode formal semantics information?Proceedings of the AAAI Conference on Artificial Intelligence, 37(11):13282– 13290. John Lyons. 1977.Semantics. Cambridge University Press. Lovish Madaan, David Esiobu, Pontus Stenetorp, Bar- bara P...
1977
-
[2024]
Publisher Copyright: © 2023 The Author(s)
Recognizing textual entailment: A review of resources, approaches, applications, and challenges.ICT Express, 10(1):132–155. Publisher Copyright: © 2023 The Author(s). Victor Raskin and Sergei Nirenburg
2023
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.