They want to pretend not to understand: The Limits of Current LLMs in Interpreting Implicit Content of Political Discourse

(2) University of Amsterdam); Alessandro Panunzi (1); Sandro Pezzelle (2) ((1) University of Florence; Walter Paci (1)

REVIEW 4 major objections 6 minor 28 references

They want to pretend not to understand: The Limits of Current LLMs in Interpreting Implicit Content of Political Discourse

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Current LLMs mostly fail to decode implicit meanings in political speech, the paper argues.

desk verdict Useful dataset, honest limits, but the open-ended 'totally correct' numbers overstate failure by treating one expert annotation as the only valid reading. read the letter →

arxiv 2506.06775 v1 pith:KFL65UXH submitted 2025-06-07 cs.CL

Walter Paci (1) , Alessandro Panunzi (1) , Sandro Pezzelle (2) ((1) University of Florence , (2) University of Amsterdam) This is my paper

classification cs.CL

keywords implicaturepresuppositionpoliticaldiscourselargelanguagemodelspragmaticsItalianchain-of-thoughtpromptingmanipulative

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that current large language models cannot reliably interpret the manipulative implicit content that pervades real political discourse. Drawing on the IMPAQTS corpus of Italian political speeches, the authors build a dataset of about 32,000 passages containing implicatures and presuppositions, each paired with an expert explanation. In a multiple-choice test, the strongest model, GPT-4o-mini, picks the right explanation 70% of the time against a 25% chance baseline and an estimated 91% expert ceiling. In an open-ended explanation task, the same model is judged fully correct only about 21% of the time with zero-shot prompting and 27% with chain-of-thought prompting. The authors conclude that these models lack the pragmatic competence needed for highly implicit language, while noting that reasoning prompts help only somewhat.

What carries the argument

The load-bearing object is IMPAQTS-PID, a new dataset built from the IMPAQTS corpus: 31,822 passages from Italian parliamentary and rally speeches, each containing an expert-annotated implicature or presupposition together with a fixed-format comment ('it implies that...' or 'it presupposes that...') and the four preceding sentences needed to interpret it. Around this dataset the paper builds two probes: a multiple-choice generation task with topic-matched distractors, and an open-ended generation task whose outputs are judged by expert linguists on a five-point scale. The chain-of-thought prompt, adapted from the corpus annotation instructions, is the mechanism that most improves open-ended performance, raising fully correct answers from 21% to 27% and lowering totally wrong answers from 39% to 33%.

What would settle it

Have independent expert linguists explain the same passages without seeing the IMPAQTS comments, then measure how often two human explanations match each other at the same 'totally correct' standard used for the models. If human-human exact-match rates are close to the model's 27% or if annotators frequently disagree about which nuance is intended, the reported failure rates would reflect task ambiguity rather than a specific LLM deficit; if human-human agreement is high while model-human agreement stays at a quarter, the conclusion stands.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a negative result with a precise magnitude: tested LLMs (GPT-4o-mini, Aya Expanse 8B, LLAMA3.1 8B, LLAMA3.2 3B) all perform above chance but far below expert level on interpreting Italian political implicatures and presuppositions. In the multiple-choice setup, all models except GPT-4o-mini fall below a simple n-gram similarity baseline, and GPT-4o-mini itself lands at 70% accuracy, more than 20 points below the bootstrap-estimated ceiling of 91%. In the open-ended setup, where the model must generate an explanation of the implied meaning, expert linguist judges rate the best model's answer as totally correct in only about one in four cases, with roughly a third totally wrong; the chain-of-thought prompt reduces but does not eliminate the failures. The paper takes this as evidence that current LLMs lack the pragmatic capacity to handle naturalistic, context-dependent political language, and identifies speaker identity, historical setting, and political affiliation as missing information that future work should supply.

Load-bearing premise

The whole evaluation assumes that each passage of implicit political content has exactly one correct explanation, the comment written by the IMPAQTS annotators, so that any model output expressing a different but plausible reading is counted as totally wrong.

Editorial extensions

If this is right

If the result generalizes, LLM-based tools that monitor political discourse cannot yet be trusted to detect or explain manipulative implicatures and presuppositions on their own.
Multiple-choice performance is inflated relative to open-ended generation, so evaluations that rely on selection tasks overestimate pragmatic understanding.
Chain-of-thought prompting is a cheap, effective lever for pragmatic tasks and worth combining with other interventions; few-shot examples did not help.
The release of IMPAQTS-PID gives the field a shared testbed for measuring progress on this task, as the paper itself releases its data and code.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the evaluation treats the single IMPAQTS annotation as the only correct reading, the reported 'totally correct' rates are likely a lower bound on genuine pragmatic competence; political implicatures often support several plausible interpretations, as the appendix's worked example itself shows.
A natural test of this bound is to measure human-human agreement on the same explanation task; if experts disagree at a rate close to the model's 'totally wrong' rate, part of the measured failure is the task's own ambiguity, not a model deficit.
The strong gap between structured pragmatic benchmark tasks and naturalistic political passages suggests that benchmark scores may not transfer to ecological discourse; a multi-genre pragmatic benchmark that includes political speech would test this directly.
Enriching prompts with speaker, party, and historical context could improve explanation quality but would also let models reproduce partisan biases from pretraining; any such intervention should be evaluated for both accuracy and fairness.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know about this paper for the dataset more than for the headline. IMPAQTS-PID, 31,822 Italian political passages with expert explanations and empirically validated context windows, is genuinely new and useful for anyone evaluating pragmatics on non-English, naturalistic data. The paper is also refreshingly honest about its own limits, which goes a long way.

The central finding — four LLMs, led by GPT4o-mini, underperform an estimated human ceiling on implicit content — is credible in direction. On the multiple-choice task, 70% vs 91% ceiling is a real gap, and only GPT4o-mini beats the BLEU-4 baseline. The authors disclose that distractor selection confounds the MCG task, so the OEG experiment carries the load. There, the stress-test concern is on the mark: the rubric instructs judges to mark outputs 'totally wrong' whenever they do not reproduce the specific IMPAQTS annotation, even if the output is a plausible reading. The worked example in Appendix C.2 makes that explicit. So the 21-27% 'totally correct' figures are really numbers for 'matches the expert annotation exactly,' not 'fails to understand the implicature.' That is a meaningful difference for political language, which is ripe with cancellable, multiply interpretable meaning. The paper should either soften the claim or run a second judge pass allowing alternative valid readings.

The other weaknesses are proportionate. The OEG uses one model, 150 samples per setting, a single annotator per item, and no inter-rater agreement; the CoT improvement from 21% to 27% has no significance test and sits within the noise floor. The CoT prompt is adapted from the same annotation instructions that produced the gold labels, so part of the gain may be the model learning the house style. All of these are disclosed in the limitations, which is in the paper's favor.

What the paper does well: the resource, the context-sufficiency validation (though with only fair-to-moderate kappas), the deterministic decoding, the released code and data, and the refusal and positional-bias analysis for the Llama models.

Who it is for: researchers building pragmatic benchmarks for non-English political discourse, and people working on evaluating LLM explanations under a strict gold standard. It deserves a serious referee and revision — the dataset and task framing are worth keeping, but the OEG conclusions need reframing and ideally a second annotation pass. I would bring it to reading group to argue about the rubric.

Referee Report

4 major / 6 minor

Summary. The paper introduces IMPAQTS-PID, a dataset of 31,822 Italian political discourse passages annotated for implicatures and presuppositions, derived from the IMPAQTS corpus. Using this resource, the authors evaluate four LLMs (GPT4o-mini, Aya Expanse 8B, Llama 3.1 8B, Llama 3.2 3B) on two tasks: a multiple-choice generation (MCG) task over four explanation candidates, and an open-ended generation (OEG) task evaluated by expert linguists. The central empirical claim is that all tested models struggle to interpret implicit content: the best MCG accuracy is 70% against an estimated ceiling of 91%, and in OEG the best model produces fully correct explanations only 21% (zero-shot) to 27% (chain-of-thought) of the time. The paper also reports that chain-of-thought prompting improves OEG performance and releases the data and code.

Significance. The work is significant as a first large-scale NLP use of the IMPAQTS corpus, a substantial and ecologically valid resource of real political speeches with expert annotations of manipulative implicit content. The two-task design, the human validation of the amount of preceding context needed, the inclusion of an n-gram baseline, and the release of data and code are clear strengths. If the quantitative claims withstand scrutiny, the paper provides useful negative evidence about LLM pragmatic competence in a naturalistic, high-stakes domain. However, the central quantitative claims currently rest on evaluation choices that need tightening: the OEG gold standard assumes a single correct interpretation, the OEG results come from one model with a small sample and single annotators, and the MCG distractor generation is acknowledged as a confound. These issues are load-bearing for the headline conclusion and should be addressed before the findings can be considered robust.

major comments (4)

[Section 4.3, Appendix C.2] The OEG evaluation treats the IMPAQTS annotation as the unique correct explanation of each implicit passage. The instructions in Appendix C.2 explicitly tell judges to assess 'whether the model has captured the nuance implied by the human annotator, not whether the generated output makes sense,' and the worked example marks a plausible model output as 'totally wrong' because it does not reproduce the specific annotation 'Italy is stalled.' For political implicatures and presuppositions, which are cancellable, context-dependent, and often admit multiple legitimate readings, this uniqueness assumption is not supported. The paper's own context-sufficiency study reports only fair-to-moderate inter-annotator agreement (Fleiss kappas 0.23–0.41, Appendix C.1), and Section 7 concedes that the open-ended evaluation 'may not fully capture the complexity of models outputs.' Since Section 5.1 acknowledges that the MCG task is confounded by distractor selection, the OEG result is the main unconfounded evidence for the central negative claim, and the 21–27% 'totally correct' figures would be overstated if some 'totally wrong' outputs are acceptable alternative readings. The authors should either provide evidence that the IMPAQTS annotations are the unique valid readings, or add an evaluation category for plausible-but-different interpretations and report how often model outputs fall into that category.
[Section 5.2, Figure 4] The OEG experiment uses a single model (GPT4o-mini), 150 samples per prompting condition, and a single expert annotator per output, with no inter-rater agreement and no significance testing. The claimed CoT improvement—'totally correct' rising from 21% to 27% and 'totally wrong' falling from 39% to 33%—may be within sampling variability; a simple two-proportion test on these sample sizes would not be significant. This weakens both the 'CoT boosts performance' claim and the use of OEG as the unconfounded confirmation that current LLMs generally fail at this task. The authors should report confidence intervals, use multiple annotators or adjudication for at least a subset, and apply an appropriate statistical test for the prompting-condition comparisons.
[Section 5.1, Figures 1–3] The MCG results are conditional on the topic-model-based distractor generation method, and the authors themselves state that 'distractor selection can significantly impact model performance, introducing confounding factors and limiting result robustness.' Only GPT4o-mini consistently beats the BLEU-4 baseline, and the reported accuracy gap between hard and easy negatives shows that the task is highly sensitive to the distractor pool. As presented, the 70% accuracy figure is therefore an estimate for one particular distractor distribution, not a general measurement of the models' ability to identify correct explanations of implicit content. The paper would be stronger if it validated distractor plausibility through human pretesting and reported human accuracy on the same MCG items, so that the ceiling comparison is a direct human–model comparison rather than an extrapolation from the context-sufficiency study.
[Section 4.3] The chain-of-thought prompt is 'adapted from the detailed instructions given to the IMPAQTS corpus annotators,' and the ground-truth labels were produced under the same annotation framework, which includes one of this paper's co-authors as a corpus co-creator. This creates a risk of circularity: the CoT condition essentially primes the model with the label-generation scheme, so the observed CoT gain may reflect alignment with the annotation instructions rather than improved pragmatic reasoning in general. An ablation with a neutral or theory-agnostic reasoning prompt, or with a prompt constructed from a different pragmatic framework, would help establish that the benefit is not an artifact of prompt–label overlap.

minor comments (6)

[Abstract and Conclusion] The phrase 'all tested models struggle' is supported by the MCG experiment for all four models, but the OEG experiment was run only with GPT4o-mini; the abstract and conclusion should be phrased to make this asymmetry explicit.
[Section 3.3] Section 3.3 says the dataset retrieves 'the preceding 4 sentences,' while Section 3.2 describes allowing annotators to see up to five left-hand sentences; please clarify whether four sentences means four context sentences plus the target sentence or four context sentences total.
[Section 5.1] The estimated ceiling of 91% is described as an 88% bootstrapped value plus an additional 3% 'standing for a random chance level accuracy in the remaining samples'; the derivation of this additive correction and the bootstrap details should be reported.
[Appendix D] The refusal rates for Llama 3.1 8B and Llama 3.2 3B are reported as 0.0015% and 0.003%, but 45 and 91 refusals out of 31,822 samples correspond to about 0.14% and 0.29%, respectively; these percentages appear to be off by two orders of magnitude.
[Section 5.2, Table 1] The statement that 'small and consistent differences' between implicatures and presuppositions are 'notably similar' is based on proportions without any uncertainty quantification; adding confidence intervals or a test would make the comparison more rigorous.
[Appendix G.1] In the description of the LLM-as-judge results, the text says 'partially correct answers decrease from 0.1% of both Zero-shot and Few-shot to 0.06% of CoT,' but Figure 6 appears to show 10% and 6%; the decimal notation should be corrected for consistency.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central result is an empirical benchmark against a published expert-annotated corpus; no prediction reduces to its inputs by construction.

full rationale

The paper's central claim—that LLMs struggle to interpret presuppositions and implicatures in Italian political discourse—is an empirical evaluation, not a derivation. The gold labels are the expert annotations of the separately published IMPAQTS corpus (Cominetti et al., 2024), and the MCG and OEG tasks measure model outputs against those published labels. Although one co-author (Panunzi) is also a co-author of the corpus paper, the annotations are external data, not a theorem or fitted parameter of the present paper; the self-citation is not load-bearing. The CoT prompt is adapted from the IMPAQTS annotation instructions, which creates a possible alignment between the prompt and the gold standard, but this affects only the interpretation of the CoT improvement; the zero-shot and few-shot conditions, which do not receive the annotation manual, show the same overall negative result. The strict 'totally wrong' criterion in Appendix C.2 (judge whether the model 'captured the nuance implied by the human annotator, not whether the generated output makes sense') is a validity concern about whether political implicatures admit only one correct reading—and the authors acknowledge the evaluation 'may not fully capture the complexity of models outputs'—but this is not circularity: matching an expert annotation is a legitimate, if conservative, operationalization of correctness. No equation or prediction is equivalent to its input by construction, and no uniqueness claim is imported from the authors' prior work.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the quality and uniqueness of the IMPAQTS annotations used as ground truth, on the sufficiency of the four-sentence context window, and on the reliability of the human evaluations of generated explanations. These are domain assumptions rather than mathematical axioms.

free parameters (3)

Estimated ceiling accuracy = 91%
Derived by bootstrapping the 126-sentence human validation study (88%) and adding 3% for chance-level guessing in remaining cases; used as the reference ceiling in MCG.
Preceding context length = 4 sentences
Fixed from the human validation study where 88% of items were judged understandable with four previous sentences; used in both MCG and OEG prompts.
Topic model number of topics = 450 plus a residual category
Chosen for distractor sampling in the MCG task; controls the hardness of negative options and hence the measured accuracy gap between hard and easy subsets.

assumptions (4)

domain assumption The IMPAQTS expert comments are the correct and unique explanations of each implicit content.
The whole evaluation labels any output that misses the annotated nuance as not totally correct, even if it is a plausible alternative reading (see Appendix C.2 example).
domain assumption Four preceding sentences provide sufficient context for understanding the implicit content.
Based on a human study with only fair-to-moderate Fleiss Kappa (0.23-0.41); the proportion of understandable items reached 88% at four sentences.
domain assumption The Gricean and Lombardi Vallauri framework of implicatures and presuppositions is accepted as the correct theory of manipulative implicit content.
The paper uses the IMPAQTS annotation scheme built on this framework without testing it against alternatives.
domain assumption A single expert judge per generated explanation provides a reliable quality score.
The paper notes that inter-rater agreement was not measurable because each of the 150 samples was evaluated by one of ten annotators.

how reviews work

0 comments

Cite this review

Pith. "Pith review of They want to pretend not to understand: The Limits of Current LLMs in Interpreting Implicit Content of Political Discourse." pith.science (2026). https://pith.science/paper/KFL65UXH

@misc{pith2026250606775,
  author       = {Pith},
  title        = {Pith review of: They want to pretend not to understand: The Limits of Current LLMs in Interpreting Implicit Content of Political Discourse},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KFL65UXH}},
  note         = {Machine review of arXiv:2506.06775}
}

read the original abstract

Implicit content plays a crucial role in political discourse, where speakers systematically employ pragmatic strategies such as implicatures and presuppositions to influence their audiences. Large Language Models (LLMs) have demonstrated strong performance in tasks requiring complex semantic and pragmatic understanding, highlighting their potential for detecting and explaining the meaning of implicit content. However, their ability to do this within political discourse remains largely underexplored. Leveraging, for the first time, the large IMPAQTS corpus, which comprises Italian political speeches with the annotation of manipulative implicit content, we propose methods to test the effectiveness of LLMs in this challenging problem. Through a multiple-choice task and an open-ended generation task, we demonstrate that all tested models struggle to interpret presuppositions and implicatures. We conclude that current LLMs lack the key pragmatic capabilities necessary for accurately interpreting highly implicit language, such as that found in political discourse. At the same time, we highlight promising trends and future directions for enhancing model performance. We release our data and code at https://github.com/WalterPaci/IMPAQTS-PID

Figures

Figures reproduced from arXiv: 2506.06775 by the authors.

**Figure 2.** Hard-negatives setting. Disaggragated accuracy scores for the MCG Task on the subset of texts with a common topic [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Easy-negatives setting. Disaggragated accuracy scores for the MCG Task on the subset of texts without a common topic. support this conclusion, we evaluate the model using IronITA (Cignarella et al., 2018), an Italian benchmark for pragmatic understanding. This dataset includes manually annotated tweets and is used for binary classification tasks, aiming to determine whether a tweet is ironic or not. While the ben… view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: OEG task. Human expert evaluation of GPT4o-mini generated answers. Results refer to 150 samples. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: Trend of implicit content comprehensibility [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

**Figure 6.** Figure 6: GPT-4o evaluation of 150 samples extracted from the GPT4o-mini OEG task. [PITH_FULL_IMAGE:figures/full_fig_p025_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

28 extracted references · 27 canonical work pages

[2]

Se Serve avere più contesto è l’opzione selezion- ata, un’altra domanda con contesto sinistro espanso (evidenziata nell’esempio seguente) appare

Serve avere più contesto. Se Serve avere più contesto è l’opzione selezion- ata, un’altra domanda con contesto sinistro espanso (evidenziata nell’esempio seguente) appare. Enunciato: "E scatta una storia di demo- nizzazione che riguarda il presidente Berlusconi, che riguarda il suo partito, che riguarda la sua gente. Un’opera di demonizzazione che non ce ...

work page
[3]

Parzialmente corretto: l’output generato dal modello (i) esplica un solo contenuto implic- ito, e tale contenuto implicito è parzialmente in linea con quello annotato dall’annotatore umano ovvero, ci può essere qualche in- terpretazione diversa o sbagliata con parte dell’esplicitazione corretta; oppure (ii) nella sintesi finale o in uno dei passaggi inter...

work page
[4]

Totalmente sbagliato: l’output generato dal modello è diverso da quanto annotato dell’annotatore umano

work page
[5]

arXiv preprint arXiv:2403.05530

Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv:2403.05530. Teun A Van Dijk. 1992. Discourse and the denial of racism. Discourse & society, 3(1):87–118. Teun A Van Dijk et al. 1997. What is political discourse analysis. Belgian journal of linguistics, 11(1):11–52. Alex Wang, Amanpreet Singh, Juli...

arXiv 1992
[9]

Il contesto è sufficiente per inferire il con- tenuto implicito

work page
[10]

English: Statement: A demonization effort that fails to take down Berlusconi, not even by using bad press, bad newspapers, think of Repubblica, but not only, also Corriere

Serve avere più contesto. English: Statement: A demonization effort that fails to take down Berlusconi, not even by using bad press, bad newspapers, think of Repubblica, but not only, also Corriere. Implied content: The demonization effort tried to take down Berlusconi

work page
[12]

If More context is needed is the chosen option, another question with extended context (high- lighted in the following example) appears

More context is needed. If More context is needed is the chosen option, another question with extended context (high- lighted in the following example) appears. Text: And so begins a demonization cam- paign targeting President Berlusconi, his party, and his supporters . A demonization effort that fails to take down Berlusconi, not even by using bad press,...

work page
[13]

The context is sufficient to infer the implicit content

work page

Show all 28 references

[14]

C.2 OEG Evaluation Instructions Italian: Grazie per il tempo che stai dedicando a questa campagna di valutazione

More context is needed. C.2 OEG Evaluation Instructions Italian: Grazie per il tempo che stai dedicando a questa campagna di valutazione. Per rispondere al sondaggio, ti preghiamo di leg- gere attentamente le istruzioni che trovi qui sotto. Ti verrà chiesto di valutare testi g...
[15]

totalmente corretto

Totalmente corretto: l’output generato dal modello esplica un solo contenuto implic- ito, e tale contenuto implicito è quello pre- sente nell’annotazione. Esso può essere leg- germente parafrasato o non contenere esatta- mente tutti i dettagli apportati dall’annotatore se ques...
[16]

Corretto tra varie opzioni: l’output gener- ato dal modello (i) esplica multipli contenuti impliciti oppure lo stesso contenuto implicito con multiple interpretazioni e una di queste è totalmente corretta; oppure, (ii) se output gen- erato dal modello, esplicitando nella rispo...
[19]

Liberi e Uguali

Risposta non fornita: L’output generato dal modello non contiene alcuna risposta, è una perifrasi che aggira la domanda o è un vero e proprio rifiuto di generare una risposta. Ecco un esempio di schermata che troverai. Testo: Diseguaglianza significa un lavoro precario, incert...
[20]

Corretto tra opzioni
[21]

Parzialmente corretto
[22]

Totalmente sbagliato
[23]

Avanti" in basso a destra per procedere al successivo testo da annotare. Puoi tornare all’item precedente clic- cando su “Indietro

Risposta non fornita Siccome ciò che deve essere valutato non è se l’output del modello ci appaia sensato ma se il mod- ello ha colto la sfumatura sottesa dall’annotatore umano, in questo caso, benché ciò che il mod- ello genera sia plausibile, quello che l’annotatore umano ha...
[24]

totally correct

Totally Correct: The output generated by the model makes explicit only one implicit content, and that implicit content is the one present in the annotation. It can be slightly paraphrased or not contain exactly all the de- tails provided by the annotator if these are not prese...
[25]

Correct among various options: The out- put generated by the model (i) makes explicit multiple implicit contents or the same implicit content with multiple interpretations, and one of these is totally correct; or (ii) the output generated by the model, by making explicit the i...
[26]

Partially Correct: The output generated by the model (i) makes explicit only one implicit content, and that implicit content is partially in line with what the human annotator anno- tated, that is, there may be some different or wrong interpretations with part of the explica- ...
[27]

Totally Wrong: The output generated by the model is different from what the human anno- tator annotated
[28]

Liberi e Uguali

Answer not given: The output generated by the model contains no answer, is a circumlo- cution that avoids the question, or is a true refusal to generate a response. Here is an example of a screen you will find. Text: Inequality means precarious, uncertain, underpaid, exploited...
[29]

Correct among options
[30]

Next" at the bottom right to proceed to the next text to be annotated. You can go back to the previous item by clicking

Answer not given Since what must be evaluated is not whether the model’s output seems sensible to us but whether the model has captured the nuance implied by the human annotator, in this case, although what the model generates is plausible, what the human anno- tator highlight...

1970
[2008]

Belgian journal of linguistics, 22(1):197–219

Explicitness, implicitness and commitment attribution: A cognitive pragmatic approach. Belgian journal of linguistics, 22(1):197–219. Renáta Németh. 2023. A scoping review on the use of natural language processing in research on political polarization: trends and research pros...

2023
[2018]

Contenuto Implicito

A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New...

2018
[2023]

Is the Pope Catholic?

"Is the Pope Catholic?" Applying Chain-of- Thought Reasoning to Understanding Conversational Implicatures. arXiv preprint arXiv:2305.13826. J Richard Landis and Gary G Koch. 1977. The mea- surement of observer agreement for categorical data. biometrics, pages 159–174. Lincan L...

1977 arXiv
[2024]

In Find- ings of the Association for Computational Linguistics: ACL 2024, pages 12075–12097, Bangkok, Thailand

PUB: A pragmatics understanding benchmark for assessing LLMs’ pragmatics capabilities. In Find- ings of the Association for Computational Linguistics: ACL 2024, pages 12075–12097, Bangkok, Thailand. Association for Computational Linguistics. PF Strawson. 1964. Identifying refe...

2024
[2025]

In Proceedings of the 31st Inter- national Conference on Computational Linguistics, pages 6782–6797, Abu Dhabi, UAE

The invalsi benchmarks: measuring the linguis- tic and mathematical understanding of large language models in Italian. In Proceedings of the 31st Inter- national Conference on Computational Linguistics, pages 6782–6797, Abu Dhabi, UAE. Association for Computational Linguistics...

2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [2]

Se Serve avere più contesto è l’opzione selezion- ata, un’altra domanda con contesto sinistro espanso (evidenziata nell’esempio seguente) appare

Serve avere più contesto. Se Serve avere più contesto è l’opzione selezion- ata, un’altra domanda con contesto sinistro espanso (evidenziata nell’esempio seguente) appare. Enunciato: "E scatta una storia di demo- nizzazione che riguarda il presidente Berlusconi, che riguarda il suo partito, che riguarda la sua gente. Un’opera di demonizzazione che non ce ...

work page

[2] [3]

Parzialmente corretto: l’output generato dal modello (i) esplica un solo contenuto implic- ito, e tale contenuto implicito è parzialmente in linea con quello annotato dall’annotatore umano ovvero, ci può essere qualche in- terpretazione diversa o sbagliata con parte dell’esplicitazione corretta; oppure (ii) nella sintesi finale o in uno dei passaggi inter...

work page

[3] [4]

Totalmente sbagliato: l’output generato dal modello è diverso da quanto annotato dell’annotatore umano

work page

[4] [5]

arXiv preprint arXiv:2403.05530

Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv:2403.05530. Teun A Van Dijk. 1992. Discourse and the denial of racism. Discourse & society, 3(1):87–118. Teun A Van Dijk et al. 1997. What is political discourse analysis. Belgian journal of linguistics, 11(1):11–52. Alex Wang, Amanpreet Singh, Juli...

arXiv 1992

[5] [9]

Il contesto è sufficiente per inferire il con- tenuto implicito

work page

[6] [10]

English: Statement: A demonization effort that fails to take down Berlusconi, not even by using bad press, bad newspapers, think of Repubblica, but not only, also Corriere

Serve avere più contesto. English: Statement: A demonization effort that fails to take down Berlusconi, not even by using bad press, bad newspapers, think of Repubblica, but not only, also Corriere. Implied content: The demonization effort tried to take down Berlusconi

work page

[7] [12]

If More context is needed is the chosen option, another question with extended context (high- lighted in the following example) appears

More context is needed. If More context is needed is the chosen option, another question with extended context (high- lighted in the following example) appears. Text: And so begins a demonization cam- paign targeting President Berlusconi, his party, and his supporters . A demonization effort that fails to take down Berlusconi, not even by using bad press,...

work page

[8] [13]

The context is sufficient to infer the implicit content

work page