REVIEW 3 major objections 4 minor 11 references
Stands to Reason: Investigating the Effect of Reasoning on Idiomaticity Detection
T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Chain-of-thought reasoning yields only small, inconsistent gains for idiomaticity detection.
desk verdict A useful, honest empirical benchmark of reasoning models for idiom detection, but the headline size-dependent conclusions rest on 5-seed point estimates with no uncertainty. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the DeepSeek-R1 distillation suite, an open-source family of reasoning models fine-tuned to emit chain-of-thought (CoT) text before a final answer, evaluated against their untuned base counterparts (Qwen2.5 and Llama-3.3) and, for the small sizes, the intermediate math-tuned checkpoints. The evaluation uses four idiomaticity detection datasets (FLUTE, SemEval 2022 Task 2a, MAGPIE, DICE) and a manual labelling rubric that scores each CoT separately for 'understanding' (whether the model gives a valid idiomatic definition) and 'reasoning' (whether it correctly disambiguates the expression in context). The same rubric underlies the paper's final experiment, where definitio
What would settle it
Take a held-out set of at least 100 MAGPIE and DICE instances per model, have multiple annotators independently label the CoTs for definition accuracy and reasoning quality, check inter-annotator agreement, and rerun the definition-prompt experiment with human-written definitions instead of the 32B model's; if agreement is low or the gains vanish, the reported understanding gap and distillation effect would not replicate.
Extended reading notes
Core claim
The paper's central claim is that reasoning—generating a chain of thought before answering—has a smaller and more varied effect on idiomaticity detection than the recent enthusiasm for reasoning models would suggest. On four benchmarks (FLUTE, SemEval 2022 Task 2a, MAGPIE, DICE), moving from base models to DeepSeek-R1 reasoning variants improves macro F1 for 14B, 32B, and 70B models by small amounts, while the 1.5B and 7B models improve only relative to the math-tuned intermediate checkpoints and still underperform their base models. Manual inspection of reasoning traces shows the divide in capability: larger models usually produce accurate idiomatic definitions, smaller models often cannot,
Load-bearing premise
The manual analysis in Section 4 rests on 30 responses per model (15 correct, 15 incorrect) annotated only by the paper's authors, without reported inter-annotator agreement; the conclusion that larger models understand idioms while smaller ones do not depends on these labels being representative and unbiased.
Editorial extensions
If this is right
- If the finding holds, adding CoT reasoning to small open-source models is not a reliable route to better idiom detection: the 1.5B and 7B reasoning models stay below their base variants on average.
- Larger reasoning models gain modestly (up to ~0.095 macro F1 on MAGPIE), so reasoning helps most when the model already has enough linguistic knowledge.
- Because reasoning quality—not definition knowledge—is the deciding factor for correctness, better context-disambiguation methods are the next bottleneck, not more idiom definitions.
- Definition prompting is a safe, sometimes helpful intervention: it improved FLUTE scores for 1.5B and 7B models and did not significantly hurt DICE, so it can be applied generally without regression.
- CoT length does not predict correctness, so users should not treat longer reasoning as a sign of better idiomaticity judgements.
Reading between the lines
- Editorial extension: the definition-distillation result could generalize into a low-cost pipeline for under-resourced languages or idiom inventories, provided the teacher model's definitions are validated by human speakers first.
- Editorial extension: the manual finding that many 'incorrect' larger-model predictions are judged valid against ambiguous gold labels suggests benchmark noise, not only model error, is at play; cleaning the datasets could change the reported ordering.
- Editorial extension: a direct test of the paper's explanation would be to give small models human-written definitions instead of model-generated ones; if gains disappear, the definitions' quality, not their source, is the operative ingredient.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper evaluates the suite of DeepSeek-R1 distilled reasoning models (1.5B–70B) on four idiomaticity detection datasets (FLUTE, SemEval 2022 Task 2a, MAGPIE, DICE), comparing them with their non-reasoning base models and, for the two smallest sizes, with intermediate math-tuned variants. Macro-F1 scores are averaged over five seeds. The main empirical claim is that the effect of reasoning on idiomaticity detection is small and varies with model size: larger models (14B, 32B, 70B) show modest improvements, while smaller models often underperform their base counterparts. The paper also reports a manual analysis of chain-of-thought outputs, arguing that larger models produce accurate idiom definitions while smaller models do not, and an experiment in which definitions generated by the 32B model are appended to the prompts of smaller models, yielding gains on FLUTE but not DICE.
Significance. If the headline results are reliable, the paper provides a useful empirical datapoint on reasoning models for idiomaticity detection and introduces a plausible knowledge-distillation idea: using definitions from a larger model to prompt smaller models. The work uses public datasets, open-source checkpoints, and a straightforward evaluation protocol, with no fitted models or circular derivations. The manual CoT analysis is a thoughtful attempt to separate definitional understanding from contextual disambiguation. However, the central size-dependence claim and the claimed significant distillation gains are not supported by the reported statistics: no standard deviations, confidence intervals, or significance tests are given for the differences that drive the conclusions. As a result, the contribution is currently exploratory and requires substantial strengthening before the claims can be accepted.
major comments (3)
- [§3, Tables 1–2] The headline conclusion—reasoning has a small, size-dependent effect—rests entirely on 5-seed mean macro-F1 differences without any reported variance or test statistics. For example, Table 2 reports +0.024 (14B), +0.017 (32B), +0.035 (70B), and −0.167 (7B) as bare means. With FLUTE’s test set of 250 examples, MAGPIE’s test split of roughly 1% of 4,840 sentences, and only five seeds, these differences are well within plausible run-to-run noise. The Limitations section acknowledges Llama-70B’s inconsistency (worse by 0.037 on SemEval English) but does not quantify uncertainty. Please report per-seed standard deviations and confidence intervals, and perform paired significance tests (e.g., bootstrap across seeds or McNemar/permutation over items). If the contrasts are not significant, the ‘small effect’ finding reduces to ‘no detectable effect,’ which is a materially weaker claim.
- [§6.1, Table 5] The caption and text claim ‘significantly improved’ results for the definition-prompting experiment, and ‘no significant difference’ for DICE and for the 14B model, yet no significance test, p-value, confidence interval, or standard deviation is reported. State the test used (paired bootstrap across the five seeds, McNemar over items, or similar) and report its outcomes. For the null results on DICE, report the smallest effect size the design could plausibly detect; otherwise the conclusion that definition prompting ‘can be applied generally without risk of regression’ is unsupported.
- [§4.1–§4.2, Figure 1, Table 3] The manual analysis uses 30 responses per model per dataset (15 correct, 15 incorrect), annotated by the three authors with no inter-annotator agreement reported. Table 3’s percentages are therefore based on 15 ‘incorrect’ predictions per row (e.g., 40% = 6 examples). The qualitative conclusion that larger models have better idiomatic understanding—and the subsequent choice of the 32B model as definition generator—rests on these scores. Please report per-label agreement (e.g., Cohen’s kappa), provide full label distributions, and ideally have annotation performed blind to model identity and correctness. At minimum, present Table 3 as illustrative and state its sample-size limitations explicitly.
minor comments (4)
- [§2.2] The GPT-4o label extraction step is used without reporting extraction accuracy or the frequency of unparseable outputs. If a large fraction of outputs require extraction, the reported scores mix model behavior with extractor behavior; please report both quantities.
- [Table 4 / §5] The correlation table reports p-values below 0.05 without describing the test or applying any multiple-testing correction. The pseudo-R² values mentioned in the text are not tabulated; please include them or remove the claim.
- [§2.1] MAGPIE’s test split is very small (1% of 4,840 sentences), which makes per-dataset macro-F1 scores noisy. This should be stated wherever MAGPIE results are interpreted, and it strengthens the need for uncertainty estimates.
- [§6, Table 6] Definition quality is illustrated by five example definitions; consider reporting automatic or manual quality statistics on the full generated set so the reader can assess the distillation source.
Circularity Check
No significant circularity: the study is an empirical evaluation on public datasets with no fitted parameters or derivation chain that reduces to its inputs.
full rationale
The paper's claims are empirical comparisons of publicly released DeepSeek-R1 distilled models against their base and intermediate variants on four idiomaticity detection datasets. There is no fitted parameter that is later renamed as a prediction, no equation in which an output variable is defined in terms of the target, and no uniqueness theorem invoked to force a choice. The only self-referential elements are (a) the authors' manual annotation of model outputs and (b) the use of definitions generated by DeepSeek-R1 32B as prompts for smaller models. Neither is load-bearing circularity: the manual analysis is a descriptive error analysis, not the source of the headline performance differences, and the definition-prompt experiment is an explicit intervention that tests whether externally supplied definitions help, with the results reported as measured outcomes rather than as a derived prediction. The datasets themselves (FLUTE, SemEval, MAGPIE, DICE) are external benchmarks, and the DICE dataset, while authored in part by the current authors, was released as a public dataset with independent labels; using it as an evaluation set does not make the evaluation circular. The headline claim that reasoning has a small and varied effect is an empirical observation from Table 2, not a consequence of a derivation. Therefore no circular step can be exhibited, and the appropriate score is 0.
Assumptions & free parameters
assumptions (5)
- domain assumption Macro F1 averaged over 5 runs is a reliable metric for these datasets
- domain assumption Q6_K_M quantization does not materially change model ranking
- domain assumption GPT-4o accurately extracts labels from messy model outputs
- domain assumption DICE test labels were not seen by DeepSeek models at training time
- domain assumption Manual annotations by the three authors provide an unbiased measure of model understanding
Cite this review
Pith. "Pith review of Stands to Reason: Investigating the Effect of Reasoning on Idiomaticity Detection." pith.science (2026). https://pith.science/paper/JKNUFLSJ
@misc{pith2026250813365,
author = {Pith},
title = {Pith review of: Stands to Reason: Investigating the Effect of Reasoning on Idiomaticity Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/JKNUFLSJ}},
note = {Machine review of arXiv:2508.13365}
}
read the original abstract
The recent trend towards utilisation of reasoning models has improved the performance of Large Language Models (LLMs) across many tasks which involve logical steps. One linguistic task that could benefit from this framing is idiomaticity detection, as a potentially idiomatic expression must first be understood before it can be disambiguated and serves as a basis for reasoning. In this paper, we explore how reasoning capabilities in LLMs affect idiomaticity detection performance and examine the effect of model size. We evaluate, as open source representative models, the suite of DeepSeek-R1 distillation models ranging from 1.5B to 70B parameters across four idiomaticity detection datasets. We find the effect of reasoning to be smaller and more varied than expected. For smaller models, producing chain-of-thought (CoT) reasoning increases performance from Math-tuned intermediate models, but not to the levels of the base models, whereas larger models (14B, 32B, and 70B) show modest improvements. Our in-depth analyses reveal that larger models demonstrate good understanding of idiomaticity, successfully producing accurate definitions of expressions, while smaller models often fail to output the actual meaning. For this reason, we also experiment with providing definitions in the prompts of smaller models, which we show can improve performance in some cases.
Reference graph
Works this paper leans on
-
[4]
ArXiv:2210.17323 [cs]. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al
-
[6]
The Llama 3 Herd of Models.arXiv preprint. ArXiv:2407.21783 [cs]. Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch
-
[7]
Faithful chain-of- thought reasoning. InThe 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP- AACL 2023). Harish Tayyar Madabushi, Edward Gow-Smith, Marcos Garcia, Carolina Scarton, Marco Idiart, and Aline Villavicencio
work page 2023
-
[8]
Number: arXiv:2204.10050 arXiv:2204.10050 [cs]
SemEval-2022 Task 2: Multi- lingual Idiomaticity Detection and Sentence Embed- ding.arXiv preprint. Number: arXiv:2204.10050 arXiv:2204.10050 [cs]. Maggie Mi, Aline Villavicencio, and Nafise Sadat Moosavi
arXiv 2022
-
[9]
Gpt-4o system card.Preprint, arXiv:2410.21276. OpenAI
-
[10]
Sign of the Times: Evaluating the use of Large Lan- guage Models for Idiomaticity Detection. InProceed- ings of the Joint Workshop on Multiword Expressions and Universal Dependencies (MWE-UD) @ LREC- COLING 2024, pages 178–187, Torino, Italia. ELRA and ICCL. Qwen
work page 2024
-
[11]
Qwen2.5 Technical Report.arXiv preprint. ArXiv:2412.15115 [cs]. R˘azvan-Alexandru Sm˘adu, David-Gabriel Ion, Dumitru- Clementin Cercel, Florin Pop, and Mihaela-Claudia Cercel
-
[12]
Investigating Large Language Mod- els for Complex Word Identification in Multilingual and Multidomain Setups. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16764–16800, Miami, Florida, USA. Association for Computational Lin- guistics. Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman
work page 2024
Show all 11 references
-
[2022]
InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7139–7159, Abu Dhabi, United Arab Emirates
FLUTE: Figurative Language Understanding through Textual Explana- tions. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7139–7159, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. DeepSeek-AI, Daya Guo...
2022
-
[2023]
arXiv preprint arXiv:2303.08774
Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Tuhin Chakrabarty, Arkadiy Saakyan, Debanjan Ghosh, and Smaranda Muresan
-
[2025]
ArXiv:2501.12948 [cs]
DeepSeek-R1: Incen- tivizing Reasoning Capability in LLMs via Reinforce- ment Learning.arXiv preprint. ArXiv:2501.12948 [cs]. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.