Pith. sign in

REVIEW 3 major objections 5 minor 51 references

Trick or Neat: Adversarial Ambiguity and Language Model Evaluation

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Language models prompted to identify ambiguous sentences perform at near-chance levels, yet linear probes on their internal token representations decode ambiguity with accuracy above 0.9, showing ambiguity information is present but not…

desk verdict A useful new ambiguity dataset with an overreaching probe claim: the near-perfect probe accuracy likely reflects noun-class memorization, not a usable ambiguity signal, and the prompt-consistency claim is contradicted by the paper's own Table 8. read the letter →

arxiv 2506.01205 v1 pith:NYQJW7YE submitted 2025-06-01 cs.CL

Antonia Karamolegkou , Oliver Eberle , Phillip Rust , Carina Kauf , Anders S{o}gaard This is my paper
classification cs.CL
keywords ambiguitydetectionadversarialdatasetlinearprobingresidualstreamrepresentationspromptingevaluationyesbiasphonologicalsyntactic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper builds AmbAdv, a dataset of 1,529 sentences that start from syntactic, lexical, and phonological ambiguities and then applies four kinds of minimal edits, namely word-order swaps, synonym replacements, random substitutions, and rhyme-based substitutions, to create ambiguous and unambiguous variants. Prompting four open-access 7-billion-parameter language models to say whether a sentence is ambiguous yields near-chance accuracy, with clear signs of a yes bias, while the same models' internal representations, read by linear probes, separate ambiguous from unambiguous sentences with accuracy above 0.9 in some token positions. The paper's central claim is that ambiguity-related information is present in the models' representations but is not reliably deployed when the models produce answers, and that some successful behavior appears to rest on memorization of publicly available example sentences.

What carries the argument

The load-bearing object is AmbAdv, an adversarial ambiguity dataset built by taking 8 syntactically, 16 lexically, and 16 phonologically ambiguous base sentences and applying four manipulations: word-order swaps, synonym substitution, random same-category substitution, and rhyme substitution. Random and rhyme substitutions are labeled unambiguous because world knowledge rules out the low-attachment reading, and the paper validates this labeling with three annotators (Cohen's kappa 0.91). The representational analysis uses linear probes, meaning logistic regression on layer-wise residual-stream vectors, at specific token roles such as the first verb, first noun, last noun, final punctuation, and the end-of-prompt token, to test whether the label can be decoded from internal states.

What would settle it

Judge the same AmbAdv sentences with annotators who are asked to list all possible interpretations without being told to apply world-knowledge constraints, or give minimal licensing contexts; if a large share of random and rhyme variants is judged ambiguous, the reported model error rates and probe accuracies are partly measuring agreement with one plausibility standard. Alternatively, probe accuracy measured on held-out sentences generated from syntactic templates that were not in the training data would settle the memorization claim.

Watch

Extended reading notes

Core claim

On a subset of 142 sentences built from one PP-attachment template, linear probes trained on residual-stream vectors of tokens such as the last noun or final punctuation decode the ambiguous or unambiguous label with accuracies up to 0.99, whereas the same models prompted with eight different templates reach at most about 0.525 accuracy on the full AmbAdv set. The paper concludes that the models encode enough information to distinguish ambiguity but fail to leverage it in generation. Disambiguation responses also show hallucinated instruments, inanimate objects acting as agents, and repetitions of the original sentence's reading after synonym edits, which the authors take as evidence of partial memorization.

Load-bearing premise

The assumption that a sentence like 'The man saw the woman with the book' is unambiguously unambiguous because the instrument reading is implausible, since annotators were given the same instruction as the default prompt and model errors are measured against the authors' plausibility judgment.

Editorial extensions

If this is right

  • Direct prompting is not a reliable way to measure what language models know about ambiguity: near-chance scores can reflect an answer-format or yes-bias problem rather than absent knowledge.
  • Evaluations of ambiguity sensitivity should account for class distribution and prompt phrasing, since one model's good score on a balanced set was produced by answering yes to almost everything.
  • Because probes can decode the label from late-layer token representations, future work could try to route that information into generation, for example through decoding-time intervention, rather than treating prompting scores as the ceiling.
  • Rhyme-based perturbations give a new handle on phonological ambiguity in text-based models, extending adversarial evaluation beyond lexical and syntactic edits.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the probe-versus-prompt gap generalizes beyond PP-attachment templates, the headline result is not that models lack ambiguity knowledge but that they lack a reliable output channel for it, which points to alignment and format robustness rather than representation learning as the bottleneck.
  • A testable extension would be to train the same probe on representations of sentences whose labels are swapped by context, such as licensing contexts that make the dress-as-instrument reading plausible, and check whether probe accuracy tracks the context-dependent gold label or the authors' plausibility judgment.
  • The memorization hypothesis could be tested directly by measuring probe accuracy on novel syntactic templates, for example 'The chef watched the waiter with the napkin,' generated from the same manipulation procedure; if probe accuracy collapses, the 0.99 figures partly reflect exposure to the source sentences.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces AmbAdv, a dataset of 1,529 English sentences covering syntactic, lexical, and phonological ambiguity with four types of adversarial modifications (word order, synonym, random, and rhyme substitutions). The authors evaluate four 7B instruction-tuned LLMs on an ambiguity-identification prompt suite of eight templates, finding generally low and uneven accuracy. In a second analysis, they train linear probes on residual-stream representations of specific token roles (first verb, first noun, last noun, last punctuation, eos) on a 142-sentence subset derived from a single template, reporting probe accuracies up to 0.99 while prompted accuracy on the same material is at most about 0.525. They conclude that ambiguity-related information is present in model representations but is not reliably leveraged in generated outputs, and they release code and data.

Significance. If its central empirical claims hold, the paper would be a useful contribution: it provides a new adversarial ambiguity benchmark, compares prompting with representation probing on the same models, and makes a falsifiable claim about a representational--behavioral gap. The strengths include the public release of code and data, the inclusion of phonological ambiguity and rhyme-based perturbations, human annotation with reported agreement, and a 5-fold probe evaluation protocol. However, the paper's headline claim about representations is conditional on ruling out a lexical confound in the probe subset, and one of the paper's summary observations about prompt consistency is contradicted by its own tables. The dataset and prompting results are still valuable, but the representational conclusion needs additional controls before it can be accepted.

major comments (3)
  1. [Representational Analysis (Section 4, Figure 3, Appendix D)] The claim that linear probes decode ambiguity with accuracy up to 0.99 is not established, because the probe subset and the label construction admit a lexical confound. The 142-sample subset consists of variations of a single template, 'The man saw the woman with X', and the ambiguous/unambiguous label is assigned according to whether X plausibly denotes an instrument of seeing. A logistic regression on the last-noun representation can therefore reach near-perfect accuracy by classifying the semantic category of X (telescope/binoculars versus book/dress), rather than by detecting ambiguity. The observation that the last noun and the punctuation/eos tokens attending to it are the strongest probe features is exactly what this confound predicts. To support the conclusion that ambiguity-related information is present but unused, the authors should report a noun-held-out split, a non-contextual or layer-0 embedding baseline, a masked-PP-noun control, or evaluation on held-out sentence templates; without at least one such control, the representational analysis does not discriminate between 'encodes ambiguity' and 'encodes noun meaning'.
  2. [Section 4, 'Ambiguity Identification', Table 3 and Table 8] The claim that 'the models demonstrate consistent performance across 8 different prompts, with an average accuracy difference of less than 0.1 points' is contradicted by the paper's own Table 8. For Qwen-2.5 on lexical ambiguity, per-prompt accuracy ranges from 0.312 to 0.617, a difference of 0.305; for Mistral-v0.3 on phonological ambiguity it ranges from 0.177 to 0.819; and the standard deviations reported in Table 3 (for example, 0.128 for Qwen-2.5 lexical and 0.197 for Mistral-v0.3 phonetic) also exceed 0.1. If 'average' refers to some aggregation across models or datasets, that aggregation should be stated explicitly; as written, this summary observation is not supported by the reported numbers and should be corrected or removed, because it feeds into the later discussion of prompt insensitivity and yes-bias.
  3. [Dataset validation (Section 3) and Appendix B] The gold-standard labels for random and rhyme substitutions are the authors' plausibility judgments: the sentences are labeled unambiguous because the instrument reading is deemed implausible, with the paper itself noting exceptions in footnote 1. Both the prompted error rates and the probe accuracies are computed against these labels, so the central empirical comparison is in part an agreement test with the authors' world-knowledge judgment rather than with an independently established ambiguity property. The high annotator agreement on random/rhyme substitutions shows that the label is reliable under the given instruction, but it does not resolve the validity concern for items such as 'The man saw the woman with the book' if a low-attachment reading remains plausible or if plausibility is context-dependent. The paper should either restrict its claims to typical plausibility-based ambiguity resolution or report an annotation protocol that collects the two readings separately and documents cases where annotators found both readings plausible.
minor comments (5)
  1. [Appendix A.1 and Table 4] The text describes phonological ambiguity as 'homonyms', but the examples in Table 6 are homophones; the terminology should be corrected.
  2. [References] Kamath et al. (2024a) and Kamath et al. (2024b) are the same paper and should be merged into a single reference.
  3. [Table 8] The column header 'A VG' is unclear and should be expanded to 'Average' or defined in the table caption.
  4. [Appendix C.2] The sentence 'We also provide a random sample of responses from Qwen-2 and Llama-3 using the disambiguation prompt in Figure 9 in Tables 9 and 10' should be rephrased to clarify that Figure 9 is the prompt template and Tables 9 and 10 contain the responses.
  5. [Figure 3] The left panel and the PCA projections in Figure 3 are visually crowded, with overlapping token labels and axis text; the figure should be redrawn for legibility.

Circularity Check

1 steps flagged · score 6.0 of 10

Probe 'ambiguity decoding' reduces to classifying the PP noun's instrument status, the same lexical property used to define the unambiguous labels.

  1. self definitional [Section 3 (Dataset validation) and Section 4 (Representational Analysis)]
    "Based on this extra-linguistic reasoning, we classified sentences with random and rhyme word substitutions as unambiguous. ... We extract layer-wise representations of specific token roles, i.e., first verb, first noun/proper noun, last noun, last punctuation, and the final end of prompt sentence representation (eos). ... We find probe accuracies of 0.9 and higher for token representations related to the last punctuation and last noun token, starting from layer 5 on."

    The unambiguous label for random/rhyme variants is assigned precisely because the PP object noun (e.g., book, dress) cannot plausibly serve as an instrument of 'saw', while ambiguous variants keep instrument nouns (telescope, binoculars, camera). The probe is then evaluated on the representation of that same last noun, so a logistic regression need only separate instrument-noun from non-instrument-noun lexical semantics, which is definitionally identical to the labeling rule. No noun-held-out split, static-embedding baseline, or masking of the PP noun is reported, so the near-perfect probe accuracy does not demonstrate representation of structural ambiguity; it recovers the very semantic property used to construct the labels.

full rationale

The prompting evaluation and dataset construction are largely self-contained and not circular: model outputs are compared to independently annotated labels, and no fitted parameter is relabeled as a prediction. The representational-analysis claim, however, reduces in part by construction. AmbAdv's syntactic labels for random/rhyme variants are assigned by whether the final PP noun plausibly denotes an instrument of 'saw'; the probe experiment then obtains its highest accuracies from the representation of that very last noun, and from eos/punctuation tokens that attend to it. A linear probe on the last noun's embedding can separate the two classes by recovering noun semantics, which is the same property used to set the labels. The paper reports no noun-held-out split, no static-embedding baseline, and no masking of the PP noun, so the conclusion that 'ambiguity-related information seems present in the models' representations' is not independent of the labeling rule. This is a partial circularity in one of the paper's two headline claims; the dataset resource and the prompted-error-rate findings retain independent content. No load-bearing self-citation was found.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no free parameters and no new theoretical entities. It relies on three domain assumptions: the world-knowledge-based labeling of random and rhyme substitutions as unambiguous, the use of human plausibility judgments as ground truth, and the representativeness of the 142-sample PP-attachment probe subset.

assumptions (3)
  • domain assumption Random and rhyme word substitutions make syntactically ambiguous sentences semantically unambiguous because the instrument reading is implausible.
    This labeling is introduced in Section 'Dataset validation' and is the basis for the gold labels used in both prompting and probe evaluations. If the low-attachment reading remains plausible for examples like 'the woman with the book', the ground-truth labels are contestable.
  • domain assumption Human world-knowledge judgments, measured by annotator agreement, define the gold standard for ambiguity.
    The paper uses three annotators on a 50% sample with Cohen's Kappa 0.91. This assumes that naive native-speaker plausibility is the correct criterion for semantic ambiguity.
  • domain assumption Linear probe accuracy on a 142-sentence subset of one PP-attachment template measures general ambiguity encoding in LLMs.
    The representational analysis is restricted to sentence (1) and its variations. The paper generalizes from this subset to 'how language models encode ambiguity at different layers', which assumes the subset is representative and not dominated by lexical regularities.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Trick or Neat: Adversarial Ambiguity and Language Model Evaluation." pith.science (2026). https://pith.science/paper/NYQJW7YE

@misc{pith2026250601205,
  author       = {Pith},
  title        = {Pith review of: Trick or Neat: Adversarial Ambiguity and Language Model Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NYQJW7YE}},
  note         = {Machine review of arXiv:2506.01205}
}
read the original abstract

Detecting ambiguity is important for language understanding, including uncertainty estimation, humour detection, and processing garden path sentences. We assess language models' sensitivity to ambiguity by introducing an adversarial ambiguity dataset that includes syntactic, lexical, and phonological ambiguities along with adversarial variations (e.g., word-order changes, synonym replacements, and random-based alterations). Our findings show that direct prompting fails to robustly identify ambiguity, while linear probes trained on model representations can decode ambiguity with high accuracy, sometimes exceeding 90\%. Our results offer insights into the prompting paradigm and how language models encode ambiguity at different layers. We release both our code and data: https://github.com/coastalcph/lm_ambiguity.

Figures

Figures reproduced from arXiv: 2506.01205 by the authors.

Figure 1
Figure 1. Results across models for an existing ambi [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Results across models after zero-shot prompting, showing the error rate of the different synonym, random, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Representational analysis of hidden state representation across layers. Linear Probe accuracies for Llama-3 [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Comparison of the ambiguous and non-ambiguous count of sentences predicted by language models [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Default1 prompt used for identifying syntactic and lexical ambiguities. [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Default2 prompt used for identifying syntactic and lexical ambiguities. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Example of the True binary type alteration prompt for identifying syntactic and lexical ambiguities. [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Example of the False binary type alteration prompt for identifying syntactic and lexical ambiguities. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Example of the prompt used for the disambiguation task. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Default1 prompt used for identifying phonological ambiguity. [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Default2 prompt used for identifying phonological ambiguity. [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Example of the True binary type alteration prompt for identifying phonological ambiguity. [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: Example of the False binary type alteration prompt for identifying phonological ambiguity. [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 14
Figure 14. Figure 14: Results across models after zero-shot prompting, showing the error rate of the different synonym, random, [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]
Figure 15
Figure 15. Figure 15: PCA projections extracted for different token roles (eos, verb, last noun, first noun) across layers for [PITH_FULL_IMAGE:figures/full_fig_p019_15.png]
Figure 16
Figure 16. Figure 16: PCA projections extracted for different token roles (eos, verb, last noun, first noun) across layers for [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

51 extracted references · 21 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Mostafa Abdou, Vinit Ravishankar, Artur Kulmizev, and Anders S gaard. 2022. https://doi.org/10.18653/v1/2022.acl-long.476 Word order does matter and shuffled language models know it . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6907--6919, Dublin, Ireland. Association for Comput...

    doi:10.18653/v1/2022.acl-long.476 2022

  4. [4]

    Dmitriy Bespalov, Sourav Bhabesh, Yi Xiang, Liutong Zhou, and Yanjun Qi. 2023. https://doi.org/10.18653/v1/2023.acl-industry.56 Towards building a robust toxicity predictor . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pages 581--598, Toronto, Canada. Association for Computational ...

    work page doi:10.18653/v1/2023.acl-industry.56 2023

  5. [5]

    Terra Blevins, Hila Gonen, and Luke Zettlemoyer. 2023. https://doi.org/10.18653/v1/2023.acl-long.367 Prompting language models for linguistic structure . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6649--6663, Toronto, Canada. Association for Computational Linguistics

    work page doi:10.18653/v1/2023.acl-long.367 2023

  6. [6]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

  7. [7]

    Alessio Buscemi and Daniele Proverbio. 2024. https://arxiv.org/abs/2402.01715 Chatgpt vs gemini vs llama on multilingual sentiment analysis . arXiv preprint

    work page Pith review arXiv 2024

  8. [8]

    James Campbell, Phillip Guo, and Richard Ren. 2023. https://openreview.net/forum?id=RDyvhOgFvQ Localizing lying in llama: Understanding instructed dishonesty on true-false questions through prompting, probing, and patching . In Socially Responsible Language Modelling Research

    work page 2023

Show all 51 references
  1. [9]

    Marianne de Heer Kloots and Willem Zuidema. 2024. https://doi.org/10.21437/Interspeech.2024-2490 Human-like linguistic biases in neural speech models: Phonetic categorization and phonotactic constraints in wav2vec2.0 . In Interspeech 2024, pages 4593--4597, Kos, Greece

    2024 doi

  2. [10]

    Vittoria Dentella, Fritz Günther, and Evelina Leivada. 2023. https://doi.org/10.1073/pnas.2309583120 Systematic testing of three language models reveals low language accuracy, absence of response stability, and a yes-response bias . Proceedings of the National Academy of Scien...

    2023 doi

  3. [11]

    Tiwalayo Eisape, Vineet Gangireddy, Roger Levy, and Yoon Kim. 2022. https://doi.org/10.18653/v1/2022.findings-emnlp.203 Probing for incremental parse states in autoregressive language models . In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2801...

    2022 doi

  4. [12]

    Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Sch \"u tze, and Yoav Goldberg. 2021. https://doi.org/10.1162/tacl_a_00410 Measuring and improving consistency in pretrained language models . Transactions of the Association for Computati...

    2021 doi

  5. [13]

    Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Ke Li, Junteng Jia, Yuan Shangguan, Jay Mahadeokar, Ozlem Kalinli, Christian Fuegen, and Mike Seltzer. 2024. https://doi.org/10.18653/v1/2024.naacl-long.309 A udio C hat L lama: Towards general-purpose speech abilities for LLM s . ...

    2024 doi

  6. [14]

    Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019. https://doi.org/10.18653/v1/N19-1004 Neural language models as psycholinguistic subjects: Representations of syntactic state . In Proceedings of the 2019 Conference of the North...

    2019 doi

  7. [15]

    Khapra, and Balaraman Ravindran

    Shreya Goyal, Sumanth Doddapaneni, Mitesh M. Khapra, and Balaraman Ravindran. 2023. https://doi.org/10.1145/3593042 A survey of adversarial defenses and robustness in nlp . ACM Comput. Surv., 55(14s)

    2023 doi

  8. [16]

    John Hewitt and Christopher D. Manning. 2019. https://doi.org/10.18653/v1/N19-1419 A structural probe for finding syntax in word representations . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Langu...

    2019 doi

  9. [17]

    Yu-Lun Hsieh, Minhao Cheng, Da-Cheng Juan, Wei Wei, Wen-Lian Hsu, and Cho-Jui Hsieh. 2019. https://doi.org/10.18653/v1/P19-1147 On the robustness of self-attentive models . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1520--...

    2019 doi

  10. [18]

    Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. https://doi.org/10.18653/v1/2020.acl-main.158 A systematic assessment of syntactic generalization in neural language models . In Proceedings of the 58th Annual Meeting of the Association for Computationa...

    2020 doi

  11. [19]

    Jennifer Hu and Roger Levy. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.306 Prompting is not a substitute for probability measurements in large language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5040--5060, ...

    2023 doi

  12. [20]

    Jennifer Hu, Kyle Mahowald, Gary Lupyan, Anna Ivanova, and Roger Levy. 2024. https://doi.org/10.1073/pnas.2400917121 Language models align with human judgments on key grammatical constructions . Proceedings of the National Academy of Sciences, 121(36):e2400917121

    2024 doi

  13. [21]

    Ivanova, Aalok Sathe, Benjamin Lipkin, Unnathi Kumar, Setayesh Radkani, Thomas H

    Anna A. Ivanova, Aalok Sathe, Benjamin Lipkin, Unnathi Kumar, Setayesh Radkani, Thomas H. Clark, Carina Kauf, Jennifer Hu, R. T. Pramod, Gabriel Grand, Vivian Paulun, Maria Ryskina, Ekin Akyürek, Ethan Wilcox, Nafisa Rashid, Leshem Choshen, Roger Levy, Evelina Fedorenko, Joshu...

    2024 arXiv

  14. [23]

    Gaurav Kamath, Sebastian Schuster, Sowmya Vajjala, and Siva Reddy. 2024 b . https://doi.org/10.1162/tacl_a_00670 Scope ambiguities in large language models . Transactions of the Association for Computational Linguistics, 12:738--754

    2024 doi

  15. [24]

    Carina Kauf, Anna A Ivanova, Giulia Rambelli, Emmanuele Chersoni, Jingyuan Selena She, Zawad Chowdhury, Evelina Fedorenko, and Alessandro Lenci. 2023. https://arxiv.org/abs/2212.01488 Event Knowledge in Large Language Models: The Gap Between the Impossible and the Unlikely . C...

    2023 arXiv

  16. [25]

    Aryan Keluskar, Amrita Bhattacharjee, and Huan Liu. 2024. https://doi.org/10.1109/BigData62323.2024.10825265 Do llms understand ambiguity in text? a case study in open-world question answering . In 2024 IEEE International Conference on Big Data (BigData), pages 7485--7490

    2024

  17. [26]

    Mary Ogbuka Kenneth, Foaad Khosmood, and Abbas Edalat. 2024. https://arxiv.org/abs/2402.01759 Systematic literature review: Computational approaches for humour style classification . arXiv preprint

    2024 arXiv

  18. [27]

    Margaret Li and Julian Michael. 2022. https://doi.org/10.18653/v1/2022.dadc-1.4 Overconfidence in the face of ambiguity with adversarial data . In Proceedings of the First Workshop on Dynamic Adversarial Data Collection, pages 30--40, Seattle, WA. Association for Computational...

    2022 doi

  19. [28]

    Yihang Li, Shuichiro Shimizu, Weiqi Gu, Chenhui Chu, and Sadao Kurohashi. 2022. https://aclanthology.org/2022.lrec-1.725/ VISA : An ambiguous subtitles dataset for visual scene-aware machine translation . In Proceedings of the Thirteenth Language Resources and Evaluation Confe...

    2022

  20. [29]

    Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. https://doi.org/10.1162/tacl_a_00115 Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies . Transactions of the Association for Computational Linguistics, 4:521--535

    2016 doi

  21. [30]

    Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah Smith, and Yejin Choi. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.51 We`re afraid language models aren`t modeling ambiguity . In Proceedings of the 2023 Conference...

    2023 doi

  22. [31]

    James A Michaelov, Seana Coulson, and Benjamin K Bergen. 2023. https://arxiv.org/abs/2301.08731 Can Peanuts Fall in Love with Distributional Semantics? In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 45, Sydney, Australia. Cognitive Science Society

    2023 arXiv

  23. [32]

    Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.466 A mbig QA : Answering ambiguous open-domain questions . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)...

    2020 doi

  24. [33]

    Kanishka Misra, Allyson Ettinger, and Kyle Mahowald. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.687 Experimental contexts can facilitate robust semantic property inference in language models, but inconsistently . In Proceedings of the 2024 Conference on Empirical Method...

    2024 doi

  25. [34]

    Hosein Mohebbi, Grzegorz Chrupa a, Willem Zuidema, and Afra Alishahi. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.513 Homophone disambiguation reveals patterns of context mixing in speech transformers . In Proceedings of the 2023 Conference on Empirical Methods in Natura...

    2023 doi

  26. [35]

    Charles Morris. 1946. https://pure.mpg.de/rest/items/item_2364496_4/component/file_2407464/content Signs Language and Behavior . Prentice-Hall, New York, USA

    1946

  27. [36]

    Max M \"u ller-Eberstein, Rob van der Goot, and Barbara Plank. 2022. https://doi.org/10.18653/v1/2022.acl-long.532 Probing for labeled dependency trees . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7...

    2022 doi

  28. [37]

    Mante S Nieuwland and Jos JA Van Berkum. 2006. https://doi.org/10.1162/jocn.2006.18.7.1098 When Peanuts Fall in Love: N400 Evidence for the Power of Discourse . Journal of Cognitive Neuroscience, 18(7):1098--1111

    2006 doi

  29. [38]

    Steven T Piantadosi, Harry Tily, and Edward Gibson. 2011. https://doi.org/10.1016/j.cognition.2011.10.004 The communicative function of ambiguity in language . Cognition, 122(3):280--291

    2011 doi

  30. [39]

    Charlotte Pouw, Marianne de Heer Kloots, Afra Alishahi, and Willem Zuidema. 2024. https://doi.org/10.1162/coli_a_00526 Perception of phonological assimilation by neural speech recognition models . Computational Linguistics, 50(3):1557--1585

    2024 doi

  31. [40]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. https://proceedings.mlr.press/v139/radford21a.html Learning transferable visual model...

    2021

  32. [41]

    Saba and Jean-Pierre Corriveau

    Walid S. Saba and Jean-Pierre Corriveau. 2001. http://www.jstor.org/stable/20016281 Plausible reasoning and the resolution of quantifier scope ambiguities . Studia Logica: An International Journal for Symbolic Logic, 67(2):271--289

    2001

  33. [42]

    Ali Babar, and Sharif Abuadbba

    Bushra Sabir, M. Ali Babar, and Sharif Abuadbba. 2023. https://arxiv.org/abs/2307.01225 Interpretability and transparency-driven detection and transformation of textual adversarial examples (it-dt) . arXiv preprint

    2023

  34. [43]

    Michael Spivey-Knowlton and Julie C Sedivy. 1995. https://doi.org/10.1016/0010-0277(94)00647-4 Resolving attachment ambiguities with multiple constraints . Cognition, 55(3):227--267

    1995 doi

  35. [44]

    Elias Stengel-Eskin, Jimena Guallar-Blasco, Yi Zhou, and Benjamin Van Durme. 2023. https://doi.org/10.18653/v1/2023.acl-long.569 Why did the chicken cross the road? rephrasing and analyzing ambiguous questions in VQA . In Proceedings of the 61st Annual Meeting of the Associati...

    2023 doi

  36. [45]

    Elias Stengel-Eskin, Kyle Rawlins, and Benjamin Van Durme. 2024. https://openreview.net/forum?id=qL9gogRepu Zero and few-shot semantic parsing with ambiguous inputs . In The Twelfth International Conference on Learning Representations

    2024

  37. [46]

    Ashima Suvarna, Harshita Khandelwal, and Nanyun Peng. 2024. https://doi.org/10.18653/v1/2024.knowllm-1.1 P honology B ench: Evaluating phonological skills of large language models . In Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024), pag...

    2024 doi

  38. [47]

    Abdul Karim Taha. 1983. https://doi.org/doi:10.1515/iral.1983.21.4.251 Types of syntactic ambiguity in english . International Review of Applied Linguistics in Language Teaching, 21(4):251--266

    1983 doi

  39. [48]

    Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. https://doi.org/10.1162/tacl_a_00321 BLiMP: The Benchmark of Linguistic Minimal Pairs for English . Transactions of the Association for Computational Linguistics, 8...

    2020 doi

  40. [49]

    Vera Zabotkina, Didier Bottineau, and Elena Boyarskaya. 2021. https://link.springer.com/chapter/10.1007/978-3-030-71637-0_24 Cognitive mechanisms of ambiguity resolution . In Advances in Cognitive Research, Artificial Intelligence and Neuroinformatics, pages 201--212, Cham. Sp...

    2021 doi

  41. [50]

    Xinyu Zhang, Hanbin Hong, Yuan Hong, Peng Huang, Binghui Wang, Zhongjie Ba, and Kui Ren. 2024. https://doi.org/10.1109/SP54263.2024.00053 Text-crs: A generalized certified robustness framework against textual adversarial attacks . In 2024 IEEE Symposium on Security and Privacy...

    2024

  42. [51]

    Yi Zhou, Xiaoqing Zheng, Cho-Jui Hsieh, Kai-Wei Chang, and Xuanjing Huang. 2021. https://doi.org/10.18653/v1/2021.acl-long.426 Defense against synonym substitution-based adversarial attacks via D irichlet neighborhood ensemble . In Proceedings of the 59th Annual Meeting of the...

    2021 doi

  43. [52]

    Xiliang Zhu, Chia-Tien Chang, Shayna Gardiner, David Rossouw, and Jonas Robertson. 2024. https://aclanthology.org/2024.unimplicit-1.3/ Resolving transcription ambiguity in S panish: A hybrid acoustic-lexical system for punctuation restoration . In Proceedings of the Third Work...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.