{"id":"4997be46-e843-4a0a-80a1-f055dd072335","arxiv_id":"2412.03242","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On a 1.8 million word English-Russian fashion corpus, ChatGPT achieves higher precision and F-measure than TBXTools and SketchEngine for terminology extraction, but lower recall.","lead":"This paper compares how well three tools, SketchEngine, TBXTools, and ChatGPT, extract fashion terms from an English-Russian magazine corpus, and checks the quality of ChatGPT's definitions. It finds ChatGPT is more precise but recalls fewer terms, while traditional tools recall more but add noise.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ChatGPT's precision advantage may reflect memorized fashion vocabulary rather than corpus-based extraction; the paper acknowledges ChatGPT sometimes ignores the input text, so Table 1 may not measure term extraction from the corpus.","rationale":"The reader's weakest assumption—gold standard completeness—is a legitimate methodological concern, but it is not the most load-bearing one. Section 4.3 shows that many of ChatGPT's false positives are genuine fashion terms missing from the gold standard, so improving the gold standard would tend to increase ChatGPT's precision, not erase its advantage. The more serious threat to the central claim is that the experiment does not control whether ChatGPT grounds its outputs in the input corpus. The authors acknowledge in Section 3 that ChatGPT sometimes ignores the given text and produces general fashion terminology, and in Section 4.4 they acknowledge that definition pages may be in ChatGPT's training data. Because the corpus itself comes from public web pages (Section 2.2), contamination is not an external accusation; it is a possibility the paper explicitly raises. Under contamination, Table 1 compares a memory-augmented system with corpus-only statistical/linguistic extractors, so the 'superior performance' conclusion is not a fair benchmark of extraction. The proposed neologism test is a concrete, low-cost way to settle this. If the concern lands, the paper's headline should be softened from a clean superiority claim to a conditional one; the reader's CONDITIONAL verdict already points in this direction, so no change in verdict is needed, though the stated rationale should be updated to emphasize the corpus-grounding control rather than gold-standard completeness alone.","tokens_in":8825,"tokens_out":6108,"duration_ms":60692,"concrete_test":"Construct a held-out corpus by inserting invented fashion terms that are absent from training data (e.g., 20 neologisms such as 'glovette' and Russian equivalents) into fashion articles, then run ChatGPT with the exact Section 3 prompt. Measure whether ChatGPT extracts the in-text invented terms and whether it outputs known fashion terms that do not occur in the text. If ChatGPT misses the inserted terms or outputs non-occurring known terms, the Table 1 advantage reflects memory rather than corpus-based extraction; if it extracts the inserted terms and suppresses absent terms, the benchmark is vindicated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that ChatGPT outperforms TBXTools and SketchEngine in precision and F-measure (Table 1). For this claim to be valid, ChatGPT must be extracting terms from the input corpus. The paper itself provides evidence against this: Section 3 reports that despite prompting for terms 'found ONLY in the given text', ChatGPT 'can sometimes deviate from the specific corpus under consideration, beginning to extract domain-specific terminology... without relying on the text provided.' Section 4.4 notes that near-identical definitions (word-level Levenshtein distances as low as 0-1) 'could suggest that the pages used to extract the reference definitions are included in ChatGPT.' Since the corpus was harvested from web pages that likely overlap ChatGPT's training data (Section 2.2), ChatGPT's outputs may be retrieved from parametric memory rather than from the supplied text. If so, the Table 1 comparison is not corpus-based term extraction but a test of memorized fashion vocabulary, giving ChatGPT an unfair advantage over corpus-only tools. The gold-standard completeness issue identified by the reader is real but cannot explain ChatGPT's high precision: incomplete gold standards penalize ChatGPT's novel-but-valid terms, so a more complete gold standard would raise its precision. The lack of corpus grounding is the load-bearing problem.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper describes an English-Russian comparable corpus of fashion texts (around 1.8 million words), a gold standard of 354 fashion terms and 60 definitions, and an evaluation of three terminology extraction tools: TBXTools, SketchEngine, and ChatGPT. The authors report precision, recall, and F-measure for each tool in Table 1 and analyze ChatGPT's generated definitions using word-level Levenshtein distance. The central claim is that ChatGPT achieves substantially higher precision and F-measure than the traditional corpus tools, while TBXTools and SketchEngine achieve higher recall. The paper also discusses the quality of ChatGPT's definitions, noting both accurate core-concept retention and occasional omissions or elaborations.","tokens_in":9048,"tokens_out":4521,"duration_ms":42422,"significance":"If the central quantitative claims held, the paper would provide valuable evidence for the practical use of LLMs in specialized bilingual terminology extraction, a timely topic with scarce systematic evaluation. The paper is transparent about several limitations, including possible overlap between the web-derived corpus and ChatGPT's training data, and the incompleteness of the gold standard. The corpus and gold standard themselves could be useful resources for the terminology and translation communities. However, the evaluation as currently designed has serious threats to validity that bear directly on the headline comparison, so the significance of the findings as reported is not yet established.","major_comments":[{"comment":"The paper does not establish that ChatGPT's term lists are grounded in the input corpus rather than in parametric memory. Section 3 states that ChatGPT 'can sometimes deviate from the specific corpus under consideration, beginning to extract domain-specific terminology—such as fashion terms—in a more general context without relying on the text provided.' Section 4.4 further observes that low Levenshtein distances 'could suggest that the pages used to extract the reference definitions are included in ChatGPT.' Since the corpus was harvested from the web and may overlap with ChatGPT's training data, the high precision values in Table 1 may reflect memorized fashion vocabulary rather than corpus-based extraction. This confound is load-bearing for the main claim that ChatGPT outperforms the other tools at extracting terms from the corpus. Please add a control, for example extracting terms from a held-out corpus (ideally with unique or synthetic terms not seen in training), or manually verify for a sample of outputs that the terms actually appear in the supplied text.","section":"Section 3 and Table 1"},{"comment":"The abstract claims that ChatGPT shows 'superior performance, maintaining or improving precision as more terms are considered.' Section 4.2, however, reports that precision increases from 25 to 100 terms and then decreases: 'If the number of terms is increased further, the pattern observed with the other term extraction methods is followed i.e. the precision decreases whilst the recall keeps increasing.' This direct contradiction undermines a central summary of the result. The abstract and the concluding sentence in Section 5 ('The performance of ChatGPT was robust across different k values') should be corrected to reflect the actual precision pattern, which improves only up to k=100 and then declines.","section":"Abstract and Section 4.2"},{"comment":"The completeness of the gold standard is load-bearing for Table 1, and the paper itself acknowledges its incompleteness: 'a large proportion of these terms are not present in our gold standard which shows one of the limitations of employing automatic evaluation metrics' (Section 4.3). Because the 354-term gold standard is used as the reference for precision, the precision values in Table 1 are systematically biased. The bias direction likely understates ChatGPT's precision (since many of its top-50 terms are valid fashion terms absent from the gold standard, as noted in Section 4.3), but the extent is unknown, and the F-measure differences could change materially if the gold standard were expanded. Please either augment the gold standard with the valid terms identified by ChatGPT (and ideally by the other tools), or supplement Table 1 with precision calculated by manual adjudication of a sample of extracted terms, so that the comparison is not an artifact of gold-standard coverage.","section":"Section 4.3 and Table 1"}],"minor_comments":[{"comment":"The representativeness claim is reported only as 'the corpora are nearing a state of representativeness.' Please provide the actual ReCor output, such as the estimated minimum number of words/texts and the observed curve, so the reader can assess the robustness of the claim.","section":"Section 2.1"},{"comment":"The number of terms extracted is given for TBXTools (approximately 10,000 in English and 5,000 in Russian after cleaning) and SketchEngine (approximately 10,000 per language), but not for ChatGPT. Please report the total number of extracted ChatGPT terms, as it is relevant for interpreting the precision-recall trade-off.","section":"Section 4.1"},{"comment":"The Levenshtein distance analysis reports averages (14.91 for English, 8.69 for Russian) and ranges, but no standard deviations or per-term distributions. Since the ranges are very wide (0-221 and 1-94), reporting the median and a boxplot or histogram would be more informative.","section":"Section 4.4"},{"comment":"The description of the gold-standard definitions is ambiguous: it says 'Sixty of these terms also had definitions in English' and that the definitions were 'automatically translated to Russian.' Please clarify whether the Russian definitions were harvested from Russian sources or only translated from English, as this affects the comparison of definition quality across languages.","section":"Section 2.3"},{"comment":"There is a citation inconsistency: the introduction cites 'Kigarriff et al, 2014' but the reference list has 'Kilgarriff, A., ...' (2014). Please correct the spelling.","section":"References"},{"comment":"The comparison in Table 1 lacks any measure of statistical significance or confidence intervals. Given the small gold standard and the large differences in the number of extracted terms, at least a simple bootstrap confidence interval for precision would help assess whether the observed gaps are meaningful.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses a useful and timely question, and the authors are candid about the limitations of their evaluation. However, the central comparison is currently undermined by two issues that must be addressed in revision: the lack of evidence that ChatGPT's outputs are grounded in the given corpus, and the acknowledged incompleteness of the gold standard. The abstract's claim about precision over increasing k is directly contradicted by the paper's own Section 4.2, which should be fixed regardless. I would also encourage the authors to state whether the corpus and gold standard will be made publicly available, as this would strengthen the reproducibility of the work. The paper may be better suited to a terminology or translation-oriented venue than to a general NLP venue, given the applied nature of the evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is worth a look, but the central quantitative claim should not be taken at face value. The genuinely new part is the English-Russian fashion corpus and the three-way comparison of TBXTools, SketchEngine, and ChatGPT, along with a detailed error analysis. The paper is clearly written, the gold standard construction is described transparently, and the top-50 error analysis is useful—especially the observation that many of ChatGPT's top terms are valid fashion terms missing from the gold standard. The authors also deserve credit for flagging some of the limitations we would otherwise have to point out ourselves.\n\nThe soft spots are significant, and the stress-test is on target. The paper's own Section 3 admits that ChatGPT sometimes ignores the input text and extracts general fashion terminology. Section 2.2 acknowledges the corpus texts may have been ingested during ChatGPT's training, and Section 4.4 notes that near-identical definitions \"could suggest\" training-data overlap. That means Table 1 is not a clean comparison of corpus-based term extraction. TBXTools and SketchEngine are stuck with the corpus; ChatGPT can quietly pull from parametric memory. If that is happening, the precision advantage is an artifact of the setup, not evidence of superior extraction skill. The incomplete gold standard cuts the other way—a fuller gold standard would likely raise ChatGPT's precision even further—so the memorization problem is the load-bearing one. The abstract also overstates the result, claiming precision \"maintains or improves\" as more terms are considered, which Section 4.2 directly contradicts (precision drops after around 100 terms). The Levenshtein-based definition evaluation is a weak proxy for definition quality and is confounded by likely memorization, though the qualitative analysis of elaboration and omission is genuinely informative. No code or data are released, which makes it hard to test alternative interpretations.\n\nWho should read this? Anyone working on terminology extraction for translation, and anyone thinking about whether LLMs can replace traditional corpus tools for glossary building. It deserves a serious referee—the topic is timely, the data are real, and the error analysis is valuable—but the paper needs major revision before the quantitative comparison can be trusted. In particular, I would want a controlled experiment with held-out or synthetic texts, or at minimum a rerun on data guaranteed to postdate ChatGPT's training cutoff, before accepting the precision numbers. As it stands, the paper is a useful cautionary case study more than a reliable benchmark.","headline":"A useful empirical starting point for LLM-based terminology extraction, but the headline precision comparison is compromised because ChatGPT may be drawing on memorized fashion vocabulary rather than the supplied corpus.","tokens_in":9548,"tokens_out":1890,"would_cite":false,"duration_ms":19282,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ChatGPT extracts fashion terms with higher precision and F-measure than SketchEngine or TBXTools on an English-Russian corpus, while those tools achieve higher recall.","keywords":["terminology extraction","ChatGPT","English-Russian fashion corpus","gold standard","precision and recall","SketchEngine","TBXTools","definition generation"],"falsifier":"Take the corpus and tools as described, extend the gold standard with the correct fashion terms ChatGPT found but the current one omits, and recompute precision for all three tools; if ChatGPT's precision and F-measure no longer dominate, the paper's central conclusion fails. A simpler version: ask independent fashion terminologists to judge the top 50 terms from each tool without the gold standard; the ranking reported here should reproduce only if the gold standard is not biased.","tokens_in":8641,"feed_emoji":"🧵","tokens_out":7236,"duration_ms":61232,"temperature":0.7,"pith_summary":"The paper sets out to compare how well ChatGPT, SketchEngine, and TBXTools build fashion terminology from a comparable English-Russian corpus of magazine and website texts, and whether ChatGPT can also supply usable definitions. On a gold standard of 354 terms, ChatGPT reaches a precision of 0.283 in English and 0.335 in Russian, and F-measures of 0.317 and 0.346, while TBXTools and SketchEngine reach higher recall but far lower precision and F-measure. ChatGPT's advantage is concentrated in the top of its ranked list: precision rises from 0.20 to 0.40 between 25 and 100 terms and then falls as the list grows. The paper further claims that ChatGPT's definitions of 60 common fashion terms are generally accurate but sometimes omit crucial specifics or add unnecessary detail. A sympathetic reader would take away that ChatGPT is a useful complement to, not a replacement for, conventional tools.","feed_headline":"ChatGPT beats standard tools at precise fashion-term extraction","feed_subtitle":"On English-Russian fashion texts, ChatGPT extracts fewer but better terms; classic tools find more but with more noise.","key_machinery":"The load-bearing object is the gold standard: a manually validated reference list of 354 fashion terms in English and Russian, harvested from online glossaries and used to score every extraction. On that reference, the paper applies the standard precision, recall, and F-measure protocol to ranked candidate lists, treating the first $k$ terms as what a terminologist would consult. For ChatGPT, the extraction mechanism is iterative prompting: the model is asked repeatedly to list fashion terms found in a given text, producing 20 to 50 terms per prompt until output becomes repetitive or drifts outside the corpus. For definitions, the comparison mechanism is word-level Levenshtein distance between ChatGPT's definitions and the reference definitions, measuring how much editing a terminologist would need.","core_discovery":"The central claim is that on a web-derived comparable corpus of English and Russian fashion texts, ChatGPT outperforms the dedicated term extraction tools on precision and F-measure for the task of building a bilingual glossary. The reported numbers are precision 0.283 in English and 0.335 in Russian for ChatGPT versus 0.022 and 0.055 for TBXTools and 0.009 and 0.008 for SketchEngine, with correspondingly higher F-measures; the traditional tools, by contrast, extract many more candidate terms and thereby reach recall of 0.629 to 0.833, while ChatGPT recall is only about 0.36. The paper also argues that ChatGPT's precision in the ranked list improves up to about 100 terms before declining, and that its top-50 output contains almost no non-fashion terms, although many correct terms are absent from the gold standard. For definitions, using word-level Levenshtein distance, ChatGPT's outputs average 14.91 token edits from the reference in English and 8.69 in Russian, retaining core meaning while sometimes adding context or dropping specific features.","pith_inferences":["Inference: The 354-term gold standard systematically disadvantages tools that surface contemporary or niche fashion terms, so ChatGPT's precision advantage may be overstated; adding its correct-but-absent top terms to the reference could shrink the gap.","Inference: A hybrid pipeline that runs SketchEngine or TBXTools for recall and uses ChatGPT to re-rank or filter the candidate list may outperform any single tool on both metrics.","Inference: The same iterative-prompting methodology could be transferred to other bilingual domains, but the authors' observation that ChatGPT sometimes draws on general world knowledge rather than the supplied text warns that its corpus fidelity is task-dependent."],"forward_implications":["A translator working with English-Russian fashion texts can use ChatGPT to get a short, highly relevant candidate glossary quickly, provided the gold-standard coverage matches the intended scope.","TBXTools and SketchEngine should be kept for recall-oriented corpus mining, where missing terms is more harmful than noise.","Because ChatGPT's precision declines after the first 100 terms, its ranked output should not be consumed beyond that point without additional filtering.","ChatGPT definitions need post-editing: in both languages they sometimes omit crucial specifics such as fabric or style descriptors, or add contextual detail that shifts the meaning."],"supporting_citations":[{"why":"Describes SketchEngine, the widely used term extraction tool that is one of the two traditional baselines.","marker":"(Kilgarriff et al. 2014)"},{"why":"Presents TBXTools, the open-source statistical and linguistic term extraction tool that is the other baseline.","marker":"(Oliver and Vàzquez 2015)"},{"why":"Prior comparison of GPT-4 with a statistical model for terminology extraction, which this paper extends with a fuller evaluation.","marker":"(Giguere et al. 2023)"},{"why":"Defines the edit-distance metric used to measure how closely ChatGPT's definitions match the reference definitions.","marker":"(Levenshtein, 1966)"},{"why":"Provides the ReCor representativeness method used to check that the compiled corpus is adequate for the domain.","marker":"(Corpas Pastor and Seghiri 2007)"},{"why":"Shows ChatGPT generating dictionary definitions, the capability that this paper evaluates for fashion terms.","marker":"(Lew, 2023)"}],"fun_headline_variants":["ChatGPT beats dedicated tools on fashion term precision","AI beats classic tools for precise fashion term extraction","Precision wins: ChatGPT outperforms term extraction tools","ChatGPT extracts cleaner fashion terms than classic tools","Fashion jargon: ChatGPT beats SketchEngine and TBXTools"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 354-term gold standard is a complete and unbiased reference for judging fashion terminology; the paper itself notes that many of ChatGPT's correct top terms are missing from it, which means precision comparisons depend heavily on the reference list's coverage.","fun_headline_variants_meta":{"raw":{"variants":["ChatGPT beats dedicated tools on fashion term precision","AI beats classic tools for precise fashion term extraction","Precision wins: ChatGPT outperforms term extraction tools","ChatGPT extracts cleaner fashion terms than classic tools","Fashion jargon: ChatGPT beats SketchEngine and TBXTools"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000162,"raw_usage":{"total_tokens":1251,"prompt_tokens":971,"completion_tokens":280,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":205}},"tokens_in":587,"tokens_out":280,"duration_ms":2862,"temperature":1.0,"reasoning_tokens":205,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:36:31.767780+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the corpus and tools as described, extend the gold standard with the correct fashion terms ChatGPT found but the current one omits, and recompute precision for all three tools; if ChatGPT's precision and F-measure no longer dominate, the paper's central conclusion fails. A simpler version: ask independent fashion terminologists to judge the top 50 terms from each tool without the gold standard; the ranking reported here should reproduce only if the gold standard is not biased.","supporting_citations":[],"review_version":1}