REVIEW 3 major objections 6 minor 2 references
Benchmarking terminology building capabilities of ChatGPT on an English-Russian Fashion Corpus
T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read ChatGPT extracts fashion terms with higher precision and F-measure than SketchEngine or TBXTools on an English-Russian corpus, while those tools achieve higher recall.
desk verdict A useful empirical starting point for LLM-based terminology extraction, but the headline precision comparison is compromised because ChatGPT may be drawing on memorized fashion vocabulary rather than the supplied corpus. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the gold standard: a manually validated reference list of 354 fashion terms in English and Russian, harvested from online glossaries and used to score every extraction. On that reference, the paper applies the standard precision, recall, and F-measure protocol to ranked candidate lists, treating the first $k$ terms as what a terminologist would consult. For ChatGPT, the extraction mechanism is iterative prompting: the model is asked repeatedly to list fashion terms found in a given text, producing 20 to 50 terms per prompt until output becomes repetitive or drifts outside the corpus. For definitions, the comparison mechanism is word-level Levenshtein distance between ChatGPT's definitions and the reference definitions, measuring how much editing a terminologist would need.
What would settle it
Take the corpus and tools as described, extend the gold standard with the correct fashion terms ChatGPT found but the current one omits, and recompute precision for all three tools; if ChatGPT's precision and F-measure no longer dominate, the paper's central conclusion fails. A simpler version: ask independent fashion terminologists to judge the top 50 terms from each tool without the gold standard; the ranking reported here should reproduce only if the gold standard is not biased.
Extended reading notes
Core claim
The central claim is that on a web-derived comparable corpus of English and Russian fashion texts, ChatGPT outperforms the dedicated term extraction tools on precision and F-measure for the task of building a bilingual glossary. The reported numbers are precision 0.283 in English and 0.335 in Russian for ChatGPT versus 0.022 and 0.055 for TBXTools and 0.009 and 0.008 for SketchEngine, with correspondingly higher F-measures; the traditional tools, by contrast, extract many more candidate terms and thereby reach recall of 0.629 to 0.833, while ChatGPT recall is only about 0.36. The paper also argues that ChatGPT's precision in the ranked list improves up to about 100 terms before declining, and that its top-50 output contains almost no non-fashion terms, although many correct terms are absent from the gold standard. For definitions, using word-level Levenshtein distance, ChatGPT's outputs average 14.91 token edits from the reference in English and 8.69 in Russian, retaining core meaning while sometimes adding context or dropping specific features.
Load-bearing premise
The load-bearing premise is that the 354-term gold standard is a complete and unbiased reference for judging fashion terminology; the paper itself notes that many of ChatGPT's correct top terms are missing from it, which means precision comparisons depend heavily on the reference list's coverage.
Editorial extensions
If this is right
- A translator working with English-Russian fashion texts can use ChatGPT to get a short, highly relevant candidate glossary quickly, provided the gold-standard coverage matches the intended scope.
- TBXTools and SketchEngine should be kept for recall-oriented corpus mining, where missing terms is more harmful than noise.
- Because ChatGPT's precision declines after the first 100 terms, its ranked output should not be consumed beyond that point without additional filtering.
- ChatGPT definitions need post-editing: in both languages they sometimes omit crucial specifics such as fabric or style descriptors, or add contextual detail that shifts the meaning.
Reading between the lines
- Inference: The 354-term gold standard systematically disadvantages tools that surface contemporary or niche fashion terms, so ChatGPT's precision advantage may be overstated; adding its correct-but-absent top terms to the reference could shrink the gap.
- Inference: A hybrid pipeline that runs SketchEngine or TBXTools for recall and uses ChatGPT to re-rank or filter the candidate list may outperform any single tool on both metrics.
- Inference: The same iterative-prompting methodology could be transferred to other bilingual domains, but the authors' observation that ChatGPT sometimes draws on general world knowledge rather than the supplied text warns that its corpus fidelity is task-dependent.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper describes an English-Russian comparable corpus of fashion texts (around 1.8 million words), a gold standard of 354 fashion terms and 60 definitions, and an evaluation of three terminology extraction tools: TBXTools, SketchEngine, and ChatGPT. The authors report precision, recall, and F-measure for each tool in Table 1 and analyze ChatGPT's generated definitions using word-level Levenshtein distance. The central claim is that ChatGPT achieves substantially higher precision and F-measure than the traditional corpus tools, while TBXTools and SketchEngine achieve higher recall. The paper also discusses the quality of ChatGPT's definitions, noting both accurate core-concept retention and occasional omissions or elaborations.
Significance. If the central quantitative claims held, the paper would provide valuable evidence for the practical use of LLMs in specialized bilingual terminology extraction, a timely topic with scarce systematic evaluation. The paper is transparent about several limitations, including possible overlap between the web-derived corpus and ChatGPT's training data, and the incompleteness of the gold standard. The corpus and gold standard themselves could be useful resources for the terminology and translation communities. However, the evaluation as currently designed has serious threats to validity that bear directly on the headline comparison, so the significance of the findings as reported is not yet established.
major comments (3)
- [Section 3 and Table 1] The paper does not establish that ChatGPT's term lists are grounded in the input corpus rather than in parametric memory. Section 3 states that ChatGPT 'can sometimes deviate from the specific corpus under consideration, beginning to extract domain-specific terminology—such as fashion terms—in a more general context without relying on the text provided.' Section 4.4 further observes that low Levenshtein distances 'could suggest that the pages used to extract the reference definitions are included in ChatGPT.' Since the corpus was harvested from the web and may overlap with ChatGPT's training data, the high precision values in Table 1 may reflect memorized fashion vocabulary rather than corpus-based extraction. This confound is load-bearing for the main claim that ChatGPT outperforms the other tools at extracting terms from the corpus. Please add a control, for example extracting terms from a held-out corpus (ideally with unique or synthetic terms not seen in training), or manually verify for a sample of outputs that the terms actually appear in the supplied text.
- [Abstract and Section 4.2] The abstract claims that ChatGPT shows 'superior performance, maintaining or improving precision as more terms are considered.' Section 4.2, however, reports that precision increases from 25 to 100 terms and then decreases: 'If the number of terms is increased further, the pattern observed with the other term extraction methods is followed i.e. the precision decreases whilst the recall keeps increasing.' This direct contradiction undermines a central summary of the result. The abstract and the concluding sentence in Section 5 ('The performance of ChatGPT was robust across different k values') should be corrected to reflect the actual precision pattern, which improves only up to k=100 and then declines.
- [Section 4.3 and Table 1] The completeness of the gold standard is load-bearing for Table 1, and the paper itself acknowledges its incompleteness: 'a large proportion of these terms are not present in our gold standard which shows one of the limitations of employing automatic evaluation metrics' (Section 4.3). Because the 354-term gold standard is used as the reference for precision, the precision values in Table 1 are systematically biased. The bias direction likely understates ChatGPT's precision (since many of its top-50 terms are valid fashion terms absent from the gold standard, as noted in Section 4.3), but the extent is unknown, and the F-measure differences could change materially if the gold standard were expanded. Please either augment the gold standard with the valid terms identified by ChatGPT (and ideally by the other tools), or supplement Table 1 with precision calculated by manual adjudication of a sample of extracted terms, so that the comparison is not an artifact of gold-standard coverage.
minor comments (6)
- [Section 2.1] The representativeness claim is reported only as 'the corpora are nearing a state of representativeness.' Please provide the actual ReCor output, such as the estimated minimum number of words/texts and the observed curve, so the reader can assess the robustness of the claim.
- [Section 4.1] The number of terms extracted is given for TBXTools (approximately 10,000 in English and 5,000 in Russian after cleaning) and SketchEngine (approximately 10,000 per language), but not for ChatGPT. Please report the total number of extracted ChatGPT terms, as it is relevant for interpreting the precision-recall trade-off.
- [Section 4.4] The Levenshtein distance analysis reports averages (14.91 for English, 8.69 for Russian) and ranges, but no standard deviations or per-term distributions. Since the ranges are very wide (0-221 and 1-94), reporting the median and a boxplot or histogram would be more informative.
- [Section 2.3] The description of the gold-standard definitions is ambiguous: it says 'Sixty of these terms also had definitions in English' and that the definitions were 'automatically translated to Russian.' Please clarify whether the Russian definitions were harvested from Russian sources or only translated from English, as this affects the comparison of definition quality across languages.
- [References] There is a citation inconsistency: the introduction cites 'Kigarriff et al, 2014' but the reference list has 'Kilgarriff, A., ...' (2014). Please correct the spelling.
- [Section 4.1] The comparison in Table 1 lacks any measure of statistical significance or confidence intervals. Given the small gold standard and the large differences in the number of extracted terms, at least a simple bootstrap confidence interval for precision would help assess whether the observed gaps are meaningful.
Circularity Check
No significant circularity; the benchmark is an empirical comparison with an external gold standard.
full rationale
The paper's central comparison is an empirical benchmark: three extraction systems are run on a fixed corpus and scored against an externally harvested gold standard. No parameter is fitted to the gold standard, and no predicted quantity is defined in terms of the metric used to evaluate it. The use of ChatGPT for corpus cleaning is preprocessing, not a source of the extraction scores; the extraction step is a separate prompt-and-collect procedure. The acknowledged possibility that ChatGPT's training data overlaps the corpus or the gold-standard web pages is a data-contamination/validity threat, not a circular reduction: the paper does not define 'accuracy' as 'similarity to training data', and the leakage is an empirical contingency, not a constructional identity. The ReCor representativeness check cites a co-author's tool, but the corpus representativeness claim is not the load-bearing derivation of the headline precision comparison, and the ReCor output ('nearing a state of representativeness') is not used to define the evaluation metric. Therefore no circular step meeting the quoted-reduction standard is present.
Assumptions & free parameters
assumptions (4)
- domain assumption The gold standard of 354 terms, harvested from web glossaries and machine-translated with manual correction, accurately reflects the full set of fashion terms in the corpus.
- domain assumption The corpus, cleaned with ChatGPT, is representative of fashion discourse in English and Russian.
- domain assumption Word-level Levenshtein distance is a meaningful measure of definition quality.
- domain assumption The reference definitions extracted from online glossaries are canonical and correct for the fashion domain.
Cite this review
Pith. "Pith review of Benchmarking terminology building capabilities of ChatGPT on an English-Russian Fashion Corpus." pith.science (2026). https://pith.science/paper/QUOKTGD7
@misc{pith2026241203242,
author = {Pith},
title = {Pith review of: Benchmarking terminology building capabilities of ChatGPT on an English-Russian Fashion Corpus},
year = {2026},
howpublished = {\url{https://pith.science/paper/QUOKTGD7}},
note = {Machine review of arXiv:2412.03242}
}
read the original abstract
This paper compares the accuracy of the terms extracted using SketchEngine, TBXTools and ChatGPT. In addition, it evaluates the quality of the definitions produced by ChatGPT for these terms. The research is carried out on a comparable corpus of fashion magazines written in English and Russian collected from the web. A gold standard for the fashion terminology was also developed by identifying web pages that can be harvested automatically and contain definitions of terms from the fashion domain in English and Russian. This gold standard was used to evaluate the quality of the extracted terms and of the definitions produced. Our evaluation shows that TBXTools and SketchEngine, while capable of high recall, suffer from reduced precision as the number of terms increases, which affects their overall performance. Conversely, ChatGPT demonstrates superior performance, maintaining or improving precision as more terms are considered. Analysis of the definitions produced by ChatGPT for 60 commonly used terms in English and Russian shows that ChatGPT maintains a reasonable level of accuracy and fidelity across languages, but sometimes the definitions in both languages miss crucial specifics and include unnecessary deviations. Our research reveals that no single tool excels universally; each has strengths suited to particular aspects of terminology extraction and application.
Reference graph
Works this paper leans on
-
[1]
Afzaal, M., Naqvi, S. B. & Qiang, G. (2023), Language, Corpora, and Technology in Applied Linguistics, Frontiers Media SA Chodkiewicz, C., Bourigault, D. and Humbley, J. (2002), ‘Making a workable glossary out of a specialised corpus’, Lexis in Contrast: Corpus-based Approaches 7, 249 Corpas Pastor, G. & Seghiri, M. (2007), ‘ El concepto de representativi...
work page 2023
-
[2024]
Journal of Translation Studies 4, 67–94
Terminology in the Age of AI: The Transformation of Terminology Theory and Practice. Journal of Translation Studies 4, 67–94. https://doi.org/10.3726/JTS012024.04 Muegge, U. (2023) Terminology Extraction for Translation and Interpretation Made Easy: How to use ChatGPT and other low -cost, web -based programs to create terminology extraction lists and glos...
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.