A Data-Driven Approach to Idiomaticity Based on Experts' Criteria in Theoretical Linguistics

Aleksander Zhmykhov; Anastasiya Drozdova; Anastasiya Vishnyakova; Elena Mikhalkova; Polina Gavin; Timofey Protasov

arxiv: 2605.19575 · v2 · pith:FZNQ26HCnew · submitted 2026-05-19 · 💻 cs.CL

A Data-Driven Approach to Idiomaticity Based on Experts' Criteria in Theoretical Linguistics

Elena Mikhalkova , Anastasiya Vishnyakova , Anastasiya Drozdova , Polina Gavin , Aleksander Zhmykhov , Timofey Protasov This is my paper

Pith reviewed 2026-05-20 05:54 UTC · model grok-4.3

classification 💻 cs.CL

keywords idiomaticitymulti-word expressionsexpert annotationlexical criteriagrammatical criteriatheoretical linguisticsdata-driven analysis

0 comments

The pith

Expert ratings of 286 multi-word expressions show none qualify as absolutely idiomatic.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper collects 286 multi-word expressions from theoretical linguistics sources and has experts annotate them using 16 lexical, grammatical, and other criteria for idiomaticity. The distribution of annotations reveals that lexical criteria exert the strongest influence on judgments, while grammatical criteria only apply under specific conditions. No expression receives ratings that would classify it as fully idiomatic across all criteria. Obsolete words and grammar further reduce the likelihood that an expression can be replaced by a single word. These patterns indicate that idiomaticity operates as a graded property shaped by particular linguistic features rather than an all-or-nothing category.

Core claim

The central claim is that there are no absolutely idiomatic expressions. Lexical criteria seem to be the most influential; grammatical criteria are bound to certain conditions; presence of obsolete words and grammar influence ability of an MWE to be replaced with one word.

What carries the argument

The 16 criteria drawn from theoretical linguistics sources, applied through expert annotation to MWEs collected from the same sources.

If this is right

Lexical criteria provide the primary signal for distinguishing degrees of idiomaticity in multi-word expressions.
Grammatical criteria only contribute to idiomaticity judgments when specific contextual conditions are met.
Expressions containing obsolete words or grammar are less likely to be replaceable by a single word.
Idiomaticity should be treated as a matter of degree in linguistic analysis rather than a binary property.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The annotation method could be adapted to train computational systems for grading idiomaticity in large text corpora.
The absence of absolute idiomaticity may affect how language models handle multi-word expressions during parsing or generation.
Testing the same criteria on expressions drawn from non-theoretical sources would check whether the patterns hold beyond the original collection.

Load-bearing premise

The 16 criteria drawn from theoretical sources are sufficient to capture the full notion of idiomaticity and that expert annotations provide a reliable, unbiased measurement of those criteria.

What would settle it

Finding even one multi-word expression that multiple independent groups of linguistics experts unanimously rate as satisfying every one of the 16 criteria would falsify the claim that no absolutely idiomatic expressions exist.

Figures

Figures reproduced from arXiv: 2605.19575 by Aleksander Zhmykhov, Anastasiya Drozdova, Anastasiya Vishnyakova, Elena Mikhalkova, Polina Gavin, Timofey Protasov.

**Figure 2.** Figure 2: Classification of MWEs in terms of their idiomaticity, by (Baldwin and Kim, 2010). proach classification of MWEs from another perspective: they enlist properties of MWEs and annotate several examples with these properties, acquiring a matrix of feature distribution from which they conclude about the probability of “MWEhood”, see fig. 2. In our opinion, this approach could be called data-driven, as class… view at source ↗

**Figure 3.** Figure 3: Number of MWEs scoring from 3 to 13 – left. Sum of scores for each of the 16 categories (ranged) – right. Replacement. Counter-intuitively, this category seems to be in a conflict with Grammatical change and Obsolescence. Often, if an MWE scores high in it, it scores low (or medium) in one or both. The example in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

**Figure 4.** Figure 4: 3D scatter plot of the sums of scores of MWEs in each category. Future work. Future work requires a larger annotated collection. The main obstacle here is that manual annotation that we performed is very time-consuming. We can see several more ways, beside the mentioned ones, of making it partially automatic, e.g. impossibility of translating an MWE word by word as well as translation with one word can be… view at source ↗

read the original abstract

The article observes data analysis of 286 multi-word expressions (MWEs) based on 16 lexical, grammatical and other criteria described in theoretical books and papers on the notion of idiomaticity. MWEs were collected from the same theoretical sources, and a set of experts in linguistics annotated them with these categories. The distribution of categories shows that there are no absolutely idiomatic expressions. Lexical criteria seem to be the most influential; grammatical criteria are bound to certain conditions; presence of obsolete words and grammar influence ability of an MWE to be replaced with one word.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper annotates 286 MWEs with 16 theoretical criteria and reports no absolute idiomatic cases, with lexical factors ranking highest, but the threshold for that claim is not clearly defined in the available description.

read the letter

Hi, the main point here is that the authors collected 286 MWEs from theoretical sources, had experts label them against 16 lexical, grammatical, and other criteria, and found that none met the bar for absolute idiomaticity. Lexical criteria showed the strongest pull, grammatical ones only under certain conditions, and obsolete words or grammar affected whether an expression could be swapped for a single word. That distributional picture is the core observation. What is new is the specific annotated set and the direct application of those exact criteria to produce influence rankings and the zero-absolute result. It is a straightforward empirical check rather than a new model or derivation. The work does a reasonable job of turning existing theoretical lists into observable patterns across real examples, which could help people building idiom tools see which distinctions actually show up in data. The soft spot is the operationalization of absolute idiomaticity. The claim that none exist rests on the annotation distributions, yet there is no stated rule for how the 16 criteria combine into that label, such as a minimum score or requiring all positive. Without that mapping, the zero count could reflect partial matches or annotation noise instead of a true absence. Details on inter-annotator agreement and exact definitions for each criterion are also missing from the summary, which leaves the influence rankings harder to trust at face value. This is for readers in computational linguistics who work on multi-word expressions or idiom detection and want a data-driven sanity check on theory. It is not a paradigm shift but supplies a usable annotated resource and some descriptive findings. I would send it to peer review so the methods section can be examined for the missing thresholds and reliability numbers.

Referee Report

2 major / 1 minor

Summary. The paper collects 286 multi-word expressions (MWEs) from theoretical linguistics sources and has linguistics experts annotate them according to 16 lexical, grammatical, and other criteria drawn from the literature. Analysis of the resulting annotation distributions supports the claims that no MWEs qualify as absolutely idiomatic, that lexical criteria exert the strongest influence, that grammatical criteria apply only under specific conditions, and that the presence of obsolete words or grammar affects an MWE's replaceability by a single word.

Significance. If the annotation protocol were fully operationalized and shown to be reliable, the work would offer a useful empirical test of theoretical criteria for idiomaticity and could inform both linguistic theory and computational models of MWEs. At present the absence of a clear mapping from the 16 criteria to the notion of 'absolute idiomaticity' and the lack of reliability metrics limit the strength of the distributional conclusions.

major comments (2)

[Abstract] Abstract: the claim that 'there are no absolutely idiomatic expressions' is not logically entailed by the reported distributions without an explicit threshold or combination rule (e.g., positive annotation on all 16 criteria, or a minimum aggregate score) that defines absolute idiomaticity. The manuscript must specify this mapping before the zero-count observation can support the stated conclusion.
[Annotation procedure] Annotation procedure (presumably §3 or §4): no information is given on inter-annotator agreement, the precise operational definitions applied to each of the 16 criteria, how disagreements were resolved, or any statistical tests used to rank criterion influence. These details are load-bearing for the reliability of the distribution claims and the assertion that lexical criteria are most influential.

minor comments (1)

[Abstract] The abstract and summary statements would benefit from a brief table or figure summarizing the 16 criteria and their observed frequencies across the 286 MWEs.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for their careful and constructive review of our manuscript. Below we provide point-by-point responses to the major comments and indicate how we plan to revise the paper.

read point-by-point responses

Referee: [Abstract] Abstract: the claim that 'there are no absolutely idiomatic expressions' is not logically entailed by the reported distributions without an explicit threshold or combination rule (e.g., positive annotation on all 16 criteria, or a minimum aggregate score) that defines absolute idiomaticity. The manuscript must specify this mapping before the zero-count observation can support the stated conclusion.

Authors: We agree that specifying the mapping from the criteria to absolute idiomaticity is essential for the validity of our conclusion. We will revise the manuscript to explicitly state that an expression is considered absolutely idiomatic only if it receives annotations indicating idiomaticity on every one of the 16 criteria. Given that our annotations of the 286 MWEs yielded no instances meeting this criterion, the data supports the claim of no absolutely idiomatic expressions. This clarification will be added to the abstract and the results section. revision: yes
Referee: [Annotation procedure] Annotation procedure (presumably §3 or §4): no information is given on inter-annotator agreement, the precise operational definitions applied to each of the 16 criteria, how disagreements were resolved, or any statistical tests used to rank criterion influence. These details are load-bearing for the reliability of the distribution claims and the assertion that lexical criteria are most influential.

Authors: The referee is correct that the annotation procedure section lacks several key details. We will expand this section in the revised manuscript to include the precise operational definitions for each of the 16 criteria, which are based directly on the theoretical linguistics literature cited in the paper. We will also detail how disagreements among the expert annotators were resolved through iterative discussions leading to consensus. Additionally, we will describe the analytical approach used to assess the relative influence of lexical versus grammatical criteria, including the distributional comparisons performed. However, formal inter-annotator agreement statistics were not computed as part of the original study, limiting our ability to report them. revision: partial

standing simulated objections not resolved

Formal inter-annotator agreement metrics

Circularity Check

0 steps flagged

Empirical annotation study with no circular derivation

full rationale

The paper selects 286 MWEs and 16 criteria from existing theoretical linguistics sources, then obtains fresh expert annotations on those criteria for the collected expressions. The central claim that no expressions are absolutely idiomatic is presented as following directly from the resulting category distributions in this new annotated dataset. No equations, fitted parameters, self-citations, or uniqueness theorems are invoked to derive the result; the observations rest on independent expert judgments rather than reducing to the paper's own inputs or prior author work by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The study rests on domain assumptions about the adequacy of the chosen criteria and the reliability of expert judgment; no numerical free parameters are fitted and no new entities are postulated.

axioms (2)

domain assumption Expert linguists can reliably and consistently apply the 16 theoretical criteria to MWEs.
The distribution results and influence rankings depend directly on the quality of the expert annotations described in the abstract.
domain assumption The 286 MWEs collected from theoretical sources form a representative sample for studying idiomaticity.
All examples originate from the same books and papers that supplied the criteria, creating a closed loop whose representativeness is not independently verified.

pith-pipeline@v0.9.0 · 5646 in / 1432 out tokens · 47887 ms · 2026-05-20T05:54:15.097589+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Foundation/AbsoluteFloorClosure.lean absolute_floor_iff_bare_distinguishability unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

The distribution of categories shows that there are no absolutely idiomatic expressions. ... vector sum ... higher is the vector sum of an annotated MWE, the more idiomatic an MWE is.

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.