REVIEW 4 major objections 4 minor 33 references
Is It JUST Semantics? A Case Study of Discourse Particle Understanding in LLMs
T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper argues that instruction-tuned LLMs, although able to separate broad uses, fail to reliably distinguish the fine-grained discourse-particle senses of the English word 'just' in naturalistic text.
desk verdict A solid, honest study of LLM sensitivity to the senses of 'just' whose main conclusion is plausible but currently over-claimed because the gold labels are not shown to be reliable. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Two probing tasks carry the argument. The first is a metalinguistic sense-labeling probe: the model assigns one of six sense labels by taking the label with the highest conditional probability after a prompt that defines each sense with examples. The second is a pairwise same-use probe: for each pair of sentences the model rates whether the two occurrences of 'just' are used the same way, and the normalized log-probability difference between 'Yes' and 'No' forms a heatmap whose block structure measures sense separation. The load-bearing resources are the two expert-labeled datasets—90 hand-constructed unambiguous sentences and 149 OpenSubtitles sentences—and the control items ('bat', 'bank', adjective and temporal senses) that show the probes can detect clear distinctions when they exist.
What would settle it
Have independent semanticists label the 90 hand-constructed and 149 subtitle sentences and calculate inter-annotator agreement; then score models against each annotator's label rather than the majority. If model accuracy equals the human annotation ceiling on ambiguous items, or if the expert labels themselves diverge on a large share of sentences, then the near-chance performance is a labeling artifact rather than a semantic deficit in the models.
Extended reading notes
Core claim
The central discovery is a measured gap between the coarse and fine senses of 'just' in LLMs. In the labeling task, models are scored by the conditional probability of each sense label; all models except the 1B one beat chance on 90 hand-built unambiguous sentences, but accuracy drops by about 0.24 on 149 subtitle sentences and does not recover when two previous utterances are added as context. In the pairwise task, the model compares sentences and gives the difference in log-probability of 'Yes' and 'No'; same-sense pairs are rated higher than different-sense pairs for every model except the smallest, but the effect sizes for the four target senses are small, while adjective, temporal, and the control words 'bat' and 'bank' separate cleanly. The authors read this as evidence that LLMs have a coarse, partially correct grasp of 'just' but have not internalized the discourse-particle distinctions that formal semantics identifies.
Load-bearing premise
The central claim rests on the assumption that every sentence in the two datasets has one correct intended sense of 'just'; if many sentences are genuinely ambiguous or the six-way taxonomy splits natural uses too finely, then low model accuracy could reflect disagreement in the labels rather than a real gap in the models' understanding.
Editorial extensions
If this is right
- Scaling model size alone will not close the gap: 70B models perform near chance on naturalistic sentences and show only weak separation of the four target discourse senses.
- Providing conversational context is not a cure; model accuracy did not improve and often fell when two prior utterances were added.
- The pairwise same-use probe is sensitive enough to detect clearly separated senses (adjective, temporal, and the control words bat and bank), so the weak signals for 'just' point to a property of the particle's semantics rather than a broken measurement.
- Systems that rely on LLMs for discourse interpretation—dialogue, summarization, reasoning about speaker intent—should expect errors where a polyfunctional particle like 'just' carries the meaning.
- Because the task uses labels from formal semantic theory, the result also constrains what 'understanding' means for function words: coarse category knowledge can coexist with missing fine-grained internalized senses.
Reading between the lines
- The result likely extends beyond 'just' to other polyfunctional discourse particles such as 'actually', 'even', 'like', and 'though'; a direct test would be to rerun the same two probes on a small inventory of such words.
- Because the paper notes that intonation often disambiguates 'just' in speech, text-only models may be operating with genuinely less information; an extension would be to test speech or prosody-aware models on the same sentences.
- The absence of a context benefit may reflect the instruction-following setup rather than the models' internal representations; a direct sentence-surprisal measure could reveal sensitivity that metalinguistic prompting misses, a caveat the paper itself flags.
- A human-model alignment analysis—scoring models by agreement with individual annotators instead of majority labels—would separate genuine semantic deficits from ambiguity in the gold standard.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates whether instruction-tuned LLMs correctly distinguish the fine-grained senses of the English discourse particle "just" (exclusive, unelaboratory, unexplanatory, emphatic, plus temporal and adjective controls). Using 90 hand-constructed sentences and 149 OpenSubtitles sentences annotated by expert semanticists, the authors run two experiments: (1) prompted sense labeling with definitions and examples, and (2) pairwise judgments of whether two occurrences of "just" are used the same way. They report that most models exceed uniform chance on the hand-constructed data but perform near the majority-class baseline on naturalistic subtitles, that context does not help, and that pairwise separation is clear only for coarser senses such as adjective and temporal. The authors conclude that LLMs have basic but incomplete knowledge of discourse-particle senses.
Significance. If the results hold, this is a valuable behavioral case study with expert-constructed and expert-annotated data, two complementary experimental paradigms, and a cross-scale model comparison that includes both open and commercial-style instruction-tuned models. The explicit release of code and data is a strength, as are the bat/bank control experiments that validate the pairwise methodology. The paper extends prior work on discourse relations to a class of understudied function words, and its negative result is falsifiable. However, the central negative claim is load-bearing on the reliability of the gold labels and on the statistical treatment of small, imbalanced samples, both of which need strengthening before the conclusions can be considered airtight.
major comments (4)
- [Section 3 and §4.2] The subtitle-label ground truth is the load-bearing premise for the central negative result, but no inter-annotator agreement statistic is reported and the adjudication procedure is underspecified: the text states that a variable subset of eight annotators labeled each sentence and that when there was disagreement, "two additional senior annotators, whose labels were both considered regardless of agreement" were used, but it never explains how a single gold label was derived from this process. With only 149 items and a skewed distribution (60 Exclusive, 33 Temporal, 22 Unexplanatory, 21 Emphatic, 12 Unelaboratory, 1 Adjective), even a modest fraction of mislabeled items materially lowers the achievable accuracy ceiling: if 20% of gold labels are wrong, the ceiling is roughly 0.8, and a model scoring 0.45 could be near that ceiling. The authors should report agreement statistics (e.g., Fleiss' kappa), per-item confidence ratings, and a clear gold-label adjudication rule, and they should make the annotator-by-item label matrix available so that the ceiling is estimable.
- [Section 5.2, Table 5] The claim that all models except Llama-3.2-1B show "significant separation" (p < .005) rests on Welch t-tests computed on 1260 same-sense pairs and 6750 different-sense pairs, but these pairs are not independent: they share common sentences, so the effective sample size is far smaller than the pairwise counts and the p-values are anti-conservative. The same issue affects the reported Cohen's d effect sizes, which are not accompanied by confidence intervals. The authors should either use a cluster-robust test or a mixed-effects model with sentence- or pair-level random effects, or a permutation test that preserves the dependence structure, and they should report confidence intervals for the effect sizes. The qualitative heatmap patterns are suggestive, but the quantitative significance claim needs a sounder statistical basis.
- [Section 4.2, Figure 2] Accuracy is reported as point estimates without confidence intervals or explicit tests against the chance baselines. For the 149-item subtitle set, the difference between a model at 0.45 and the majority-class baseline of 0.403 may not be meaningful, and the claim that models are "at chance" for several conditions should be supported by binomial or bootstrap confidence intervals per model and per condition. Without such intervals, it is impossible to judge which model-by-condition differences are reliable, and the pattern of "near chance" on subtitles versus above-chance on hand-constructed data is not quantified in a way that supports the paper's central conclusion.
- [Section 3 and §5.2] The hand-constructed sentences are described as having been "carefully created by an expert" who is a graduate linguist and native speaker, but no second annotation, adjudication, or agreement measure is reported for these 90 items. Since the pairwise experiment in §5.2 relies entirely on these labels as ground truth, any ambiguous or mislabeled items would attenuate the measured same-sense versus different-sense separation, particularly for the fine-grained target senses that are the focus of the paper. A validation pass by a second annotator, or at least a public release of the items with confidence ratings, would strengthen the interpretation of the effect sizes in Table 5.
minor comments (4)
- [Appendix F, Figure 6] The sense-labeling prompt uses the label "Exclusionary" for what the rest of the paper calls "Exclusive"; the authors should harmonize this terminology to avoid confusion in reproducing the exact prompt.
- [Section 3] The sentence "strong speaker consensus on the reading of an occurrence of just does remove more ambiguous sentences from out data" contains a typo: "out data" should be "our data."
- [Section 5.2 and Table 5] The notation is inconsistent across model names (e.g., "Llama3.2 1B" in Table 5 versus "Llama-3.2-1b" in the text and figures); the authors should standardize model identifiers.
- [Figure 3] The "ideal" heatmap is embedded in the same row as the model heatmaps, which makes it easy to mistake it for a model output; placing it separately or adding a clear panel label would improve readability.
Circularity Check
No significant circularity: the paper's negative result is an empirical evaluation against external ground-truth labels; no prediction reduces to a fitted input or to a self-citation chain.
full rationale
The paper's central claim—that instruction-tuned LLMs only partially separate fine-grained senses of 'just'—is an empirical measurement, not a derivation. The sense taxonomy is taken from published formal-semantics literature (Lee, Warstadt, Beltrama, Coppock & Beaver), and the models' outputs are compared against expert-created ground-truth labels; the outcome (near-chance accuracy on subtitle items) is not built into the experimental setup, since the models could in principle have matched the labels. There are no fitted parameters, and no 'prediction' is a renamed input. The bat/bank controls provide an external check that the pairwise method can detect sense distinctions, which strengthens the conclusion rather than presupposing it. The only author-overlap citation that touches the design is Deo and Thomas (2025), a unified account of 'just' whose taxonomy is cited as background; the four target senses are additionally sourced to independent prior work (Coppock & Beaver 2014; Warstadt 2020; Wiegand 2018; Windhearn 2021; Lee 1987; Beltrama 2018, 2022), so the self-citation is not load-bearing. The reported absence of inter-annotator agreement on the subtitle labels is a validity concern about ground-truth reliability, not a circularity: labeling noise would depress accuracy for all models but is not a logical reduction of the paper's claim to its inputs. Score 1 reflects the minor background self-citation; there is no circular step.
Assumptions & free parameters
assumptions (5)
- domain assumption The six-way taxonomy of just senses (exclusive, unelaboratory, unexplanatory, emphatic, temporal, adjective) is a valid and applicable scheme for classifying occurrences of just.
- domain assumption Hand-constructed sentences have a single strong primary reading without context.
- domain assumption Log-probability of label continuations measures the model's metalinguistic sense knowledge.
- domain assumption Subtitle annotations by volunteer semanticists reflect consensus ground truth.
- domain assumption In the pairwise test, 'Yes/No' judgments measure sense identity rather than general sentence similarity.
Cite this review
Pith. "Pith review of Is It JUST Semantics? A Case Study of Discourse Particle Understanding in LLMs." pith.science (2026). https://pith.science/paper/O5K4FYWI
@misc{pith2026250604534,
author = {Pith},
title = {Pith review of: Is It JUST Semantics? A Case Study of Discourse Particle Understanding in LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/O5K4FYWI}},
note = {Machine review of arXiv:2506.04534}
}
read the original abstract
Discourse particles are crucial elements that subtly shape the meaning of text. These words, often polyfunctional, give rise to nuanced and often quite disparate semantic/discourse effects, as exemplified by the diverse uses of the particle "just" (e.g., exclusive, temporal, emphatic). This work investigates the capacity of LLMs to distinguish the fine-grained senses of English "just", a well-studied example in formal semantics, using data meticulously created and labeled by expert linguists. Our findings reveal that while LLMs exhibit some ability to differentiate between broader categories, they struggle to fully capture more subtle nuances, highlighting a gap in their understanding of discourse particles.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card
2024
- [2]
-
[3]
Andrea Beltrama. 2022. Just perfect, simply the best: an analysis of emphatic exclusion. Linguistics and Philosophy, 45(2):321--364
work page 2022
-
[4]
Anne Beyer, Sharid Lo \'a iciga, and David Schlangen. 2021. https://doi.org/10.18653/v1/2021.naacl-main.328 Is incoherence surprising? targeted evaluation of coherence prediction from language models . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 41...
-
[5]
Andrea Bonomi and Paolo Casalegno. 1993. Only: Association with focus in event semantics. Natural Language Semantics, 2(1):1--45
work page 1993
-
[6]
Chunkit Chan, Cheng Jiayang, Weiqi Wang, Yuxin Jiang, Tianqing Fang, Xin Liu, and Yangqiu Song. 2024. https://aclanthology.org/2024.findings-eacl.47/ Exploring the potential of C hat GPT on sentence level relations: A focus on temporal, causal, and discourse relations . In Findings of the Association for Computational Linguistics: EACL 2024, pages 684--72...
work page 2024
-
[7]
Jacob Cohen. 1988. Statistical Power Analysis for the Behavioral Sciences , 2 edition. L. Erlbaum Associates
work page 1988
-
[8]
Yan Cong, Emmanuele Chersoni, Yu-Yin Hsu, and Philippe Blache. 2023. https://doi.org/10.18653/v1/2023.blackboxnlp-1.17 Investigating the effect of discourse connectives on transformer surprisal: Language models understand connectives, E ven so they are surprised . In Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks f...
Show all 33 references
-
[9]
Elizabeth Coppock and David I Beaver. 2014. Principles of the exclusive muddle. Journal of Semantics, 31(3):371--432
2014
-
[10]
Ashwini Deo and William Carl Thomas. 2025. Addressing the widest answerable question: English “just” as a domain widening strategy. Journal of Semantics, page ffae015
2025
-
[11]
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel,...
2024
-
[12]
Patrick Georg Grosz. 2012. On the grammar of optative constructions
2012
-
[13]
Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. https://doi.org/10.18653/v1/2020.acl-main.158 A systematic assessment of syntactic generalization in neural language models . In Proceedings of the 58th Annual Meeting of the Association for Computationa...
2020 doi
-
[14]
Jennifer Hu and Roger Levy. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.306 Prompting is not a substitute for probability measurements in large language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5040--5060, ...
2023 doi
-
[15]
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv:2310.06825
2023 arXiv
-
[16]
Najoung Kim, Roma Patel, Adam Poliak, Patrick Xia, Alex Wang, Tom McCoy, Ian Tenney, Alexis Ross, Tal Linzen, Benjamin Van Durme, et al. 2019. Probing what different nlp tasks teach machines about function word comprehension. In Proceedings of the Eighth Joint Conference on Le...
2019
-
[17]
David Lee. 1987. The semantics of just. Journal of pragmatics, 11(3):377--398
1987
-
[18]
Pierre Lison, J \"o rg Tiedemann, and Milen Kouylekov. 2018. Opensubtitles2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)
2018
-
[19]
Kanishka Misra. 2022. https://arxiv.org/abs/2203.13112 minicons: Enabling flexible behavioral and representational analyses of transformer language models . arXiv:2203.13112
2022 arXiv
-
[20]
Kanishka Misra, Julia Rayz, and Allyson Ettinger. 2023. https://doi.org/10.18653/v1/2023.eacl-main.213 COMPS : Conceptual minimal pair sentences for testing robust property knowledge and its inheritance in pre-trained language models . In Proceedings of the 17th Conference of ...
2023 doi
-
[21]
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Mi...
2024 arXiv
-
[22]
Lalchand Pandia, Yan Cong, and Allyson Ettinger. 2021. https://doi.org/10.18653/v1/2021.conll-1.29 Pragmatic competence of pre-trained language models through the lens of discourse connectives . In Proceedings of the 25th Conference on Computational Natural Language Learning, ...
2021 doi
-
[23]
Gemma Team. 2024. https://doi.org/10.34740/KAGGLE/M/3301 Gemma
2024 doi
-
[24]
Alex Warstadt. 2020. "just" don’t ask: Exclusives and potential questions. In Proceedings of Sinn und Bedeutung, volume 24, pages 373--390
2020
-
[25]
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. https://doi.org/10.1162/tacl_a_00321 BL i MP : The benchmark of linguistic minimal pairs for E nglish . Transactions of the Association for Computational Linguistic...
2020 doi
-
[26]
Kangda Wei, Aayush Gautam, and Ruihong Huang. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.1 Are LLM s good annotators for discourse-level event relation extraction? In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1--19, Miami, Florida,...
2024 doi
-
[27]
Bernard L Welch. 1947. The generalization of ‘student's’problem when several different population varlances are involved. Biometrika, 34(1-2):28--35
1947
-
[28]
Mia Wiegand. 2018. Exclusive morphosemantics: Just and covert quantification. In Proceedings of the west coast conference on formal linguistics (WCCFL), volume 35, pages 419--429
2018
-
[29]
Mia Windhearn. 2021. Alternatives, exclusivity and underspecification
2021
-
[30]
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771
2019 arXiv
-
[31]
Frances Yung, Mansoor Ahmad, Merel Scholman, and Vera Demberg. 2024. https://aclanthology.org/2024.law-1.15/ Prompting implicit discourse relation annotation . In Proceedings of the 18th Linguistic Annotation Workshop (LAW-XVIII), pages 150--165, St. Julians, Malta. Associatio...
2024
-
[32]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...
-
[33]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.