{"id":"5b2c3901-07b2-4de5-a0cc-029019877dfb","arxiv_id":"2412.04509","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A two-call prompt that adds pragmatic analysis and reflection improves GPT-4o's sarcasm detection on MUStARD and SemEval2018, but the effect is not consistent across models and lacks statistical validation.","lead":"This paper introduces a two-stage prompt, PMP, that asks large language models to analyze pragmatic cues such as implicature, presupposition, and intent, then reflect before labeling a statement as sarcastic. On two sarcasm benchmarks, GPT-4o with PMP reaches higher accuracy and F1 than several chain-of-thought style prompts, although the gains are inconsistent across models and statistical significance is never tested.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim depends on SarcasmCue baseline numbers that were copied rather than rerun; duplicated BoC/GoC rows in Table 1 show those baselines are unreliable, so the comparison is not established.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing point: the SOTA claim depends on comparability of SarcasmCue baselines, and the paper provides no independent verification of those numbers. My stress-test confirms this is the right concern. The duplicated BoC/GoC rows for Claude 3.5 Sonnet and LLaMA-3-8B are an internal red flag that the baseline table was assembled carelessly; because those baselines are the direct competitor for the SOTA claim, the central empirical conclusion cannot be accepted on the evidence presented. I found no deeper issue that would overturn the conditional verdict; the prompt design is plausible and the reported PMP numbers are internally consistent. The recommended resolution is therefore the same as the reader's: conditional acceptance pending reruns of the baselines under identical conditions and correction of the table. The concrete test above would settle the concern by recomputing the key baseline rows rather than relying on copied numbers. This is a good-faith concern about experimental methodology, not a claim that the results are fabricated; a copy-paste error or a subtle protocol mismatch would produce the same symptoms.","tokens_in":8752,"tokens_out":5895,"duration_ms":54960,"concrete_test":"Rerun BoC, CoC, GoC, and PMP with GPT-4o on the same MUStARD and SemEval2018 splits used in the paper, using the exact SarcasmCue prompts and identical decoding settings (e.g., temperature 0, same max tokens, same answer parser). If the rerun GoC accuracy on SemEval2018 is within noise of PMP's 86.68, or exceeds it, the SOTA claim fails. As a sanity check, also rerun Claude 3.5 Sonnet and LLaMA-3-8B with BoC and GoC: if the duplicate rows in Table 1 do not reproduce, the published baseline table is unreliable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — PMP achieves state-of-the-art performance on GPT-4o on MUStARD and SemEval2018 — is an empirical comparison claim. To support it, the SarcasmCue baselines (especially Graph of Cues, the prior SOTA prompt) must be measured under the same protocol as PMP. Section 4.3.1 states that Yao et al. had not released code, so the authors copied the SarcasmCue numbers from that paper rather than running those prompts themselves. That is an assumption of comparability: identical prompts, decoding parameters, data splits, pre-processing, and metric calculation. The assumption is already contradicted inside the paper: in Table 1, Claude 3.5 Sonnet (GoC) and LLaMA-3-8B (GoC) are reported as exactly 57.33/57.24 accuracy/F1 on SemEval2018 and 52.77/52.67 on MUStARD, and the BoC rows are also identical (65.94/65.50 and 59.71/56.70) for the two models. Two different models producing identical results on both datasets to two decimal places is not credible unless the rows were copied or the baselines are model-independent; either way the table cannot support a model-specific SOTA comparison. If the true GoC numbers under PMP's evaluation protocol differ, the large GPT-4o gap (86.68 vs 74.03 on SemEval2018) may be an artifact of evaluation conditions. The paper also reports no confidence intervals or multiple runs, so the central comparison rests entirely on these uncorroborated baseline entries. This is a correctness risk in the empirical argument, not a disagreement about method plausibility.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Pragmatic Metacognitive Prompting (PMP), a two-stage prompting method for LLM sarcasm detection that combines pragmatic analysis (implicature, presupposition, speaker intent, polarity, pretense, and literal/implied meaning) with a metacognitive reflection step. The method is evaluated on MUStARD and SemEval2018 Task 3 using GPT-4o, GPT-4o-mini, Claude 3.5 Sonnet, LLaMA-3-8B, and LLaMA-3-70B, and is compared with popular prompting methods and with the SarcasmCue baselines (BoC, CoC, GoC, ToC). The central claim is that PMP achieves state-of-the-art performance on GPT-4o on both benchmarks, with the strongest reported result being 86.68 accuracy / 83.18 Macro-F1 on SemEval2018 versus 74.03 / 74.02 for Graph of Cues. The paper also discusses limitations regarding cultural and linguistic coverage of prompting approaches.","tokens_in":9087,"tokens_out":5283,"duration_ms":45800,"significance":"If the empirical results are confirmed under a consistent evaluation protocol, PMP would be a useful and linguistically motivated extension of Metacognitive Prompting, showing that integrating pragmatic theories into the reasoning loop can substantially improve LLM sarcasm detection. The paper includes a clear connection to established pragmatic frameworks and evaluates across multiple models and prompting strategies. The main contribution is empirical, however, and its credibility currently rests on comparisons to baselines that were not fully re-run or fully documented. The manuscript also ships code via an anonymous link, which is a positive step toward reproducibility, but the prompt templates in the appendix are paraphrased rather than verbatim, and no statistical reliability measures are reported.","major_comments":[{"comment":"The central SOTA comparison against SarcasmCue is not established because the BoC, GoC, and ToC numbers are not measured under the same protocol. The paper states that Yao et al.'s code was unavailable and that results were taken from their paper. However, Table 1 reports identical BoC and GoC values for Claude 3.5 Sonnet and LLaMA-3-8B: BoC is 65.94/65.50 on SemEval2018 and 59.71/56.70 on MUStARD for both models, and GoC is 57.33/57.24 and 52.77/52.67 (with a typo '52.7' in the LLaMA row). Two distinct models producing identical results to two decimal places is implausible unless the rows were copied in error. This undermines the reliability of the baseline comparison. Please rerun these baselines under the PMP evaluation protocol, or clearly restrict the SOTA claim to the baselines that were actually run in this work.","section":"Section 4.3.1, Table 1"},{"comment":"The ToC row is reported (e.g., LLaMA-3-8B ToC: 68.88/68.21 on SemEval2018 and 61.26/58.03 on MUStARD) even though ToC is described as requiring explicit model training. No training data, hyperparameters, model architecture details, or training code are provided. If these numbers are copied from Yao et al., that should be stated explicitly; if the authors trained ToC, full implementation details must be supplied for reproducibility. As written, the comparison to ToC is not verifiable.","section":"Section 4.3.1, Table 1, Appendix A.1.4"},{"comment":"All results are single-run point estimates without variance, confidence intervals, or statistical significance tests. The SOTA claim for GPT-4o rests on a 12.65-point accuracy gap (86.68 vs 74.03) on SemEval2018 against a baseline that was not run under the same protocol. Please report multiple runs or seeds with means and standard deviations, and include a paired test (e.g., McNemar) on instance-level predictions for the primary GPT-4o comparison, given that the dataset sizes are modest.","section":"Section 5, Table 1"},{"comment":"The conclusion states that PMP 'is competitive with or beats the current state-of-the-art methods ... such as GPT4o and LLaMA-3-8B,' but Table 1 shows that on MUStARD with LLaMA-3-8B, PMP underperforms most listed baselines (e.g., PMP 53.48/54.69 versus BoC 65.94/65.50 and GoC 57.33/57.24; and PMP 53.48/54.69 versus ToC 61.26/58.03). This overstates the findings and should be revised to accurately reflect that the strong result is specific to GPT-4o.","section":"Section 6, Conclusion, and Table 1"}],"minor_comments":[{"comment":"The heading 'Presuppositon' contains a typo and should be 'Presupposition'; similar typos appear elsewhere ('statment', 'lingustic', 'explaination', 'reassesment').","section":"Section 2.1.2"},{"comment":"The text contains unresolved references 'Figure??' and 'Figure ??' in Section 3 and in the description of Figure 1; these should be replaced with the correct figure numbers.","section":"Section 3 and Figure captions"},{"comment":"Section 5 references 'Table 2' when discussing the zero-shot Claude 3.5 Sonnet result on MUStARD, but the manuscript contains only Table 1; either add the missing table or update the citation.","section":"Section 5"},{"comment":"The text mentions 'Sarcasm Corpus V1' in the Datasets paragraph, but this dataset is not among the benchmarks described in Section 4.1 (MUStARD and SemEval2018 Task 3); please clarify or correct.","section":"Section 5, Datasets paragraph"},{"comment":"Appendix A provides paraphrased prompt descriptions rather than verbatim prompt templates; for reproducibility, include the exact prompts used for both PMP calls and for each dataset variation.","section":"Appendix A"},{"comment":"The LLaMA-3-8B GoC row on MUStARD reports accuracy as '52.7' rather than '52.77', which is inconsistent with the corresponding Claude 3.5 Sonnet row and should be corrected.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be an early-stage draft with missing table references, unresolved figure citations, and a data-integrity issue in Table 1 (duplicate BoC/GoC rows across models). The central empirical claim is potentially interesting, but the baseline comparison and the ToC results need to be substantially reworked. I would be willing to review a revised version that reruns baselines under a consistent protocol, adds statistical reliability measures, and fixes the completeness issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: PMP is a real new prompt, and the GPT-4o numbers look promising, but the SOTA claim does not survive contact with Table 1. Claude 3.5 Sonnet and LLaMA-3-8B are listed with identical GoC and BoC rows to two decimal places on both datasets. Two different models do not produce identical numbers; those rows were copied. Section 4.3.1 admits the SarcasmCue baselines were transcribed from Yao et al. rather than rerun under the same protocol. That makes the headline result (PMP 86.68 vs GoC 74.03 on SemEval2018) an artifact of evaluation conditions until proven otherwise.\n\nWhat is genuinely new: the PMP prompt itself. It is a two-stage design — first analyze implicature, presupposition, intent, polarity, pretense, and literal-vs-implied meaning; then reflect and give a verdict. That is a clean, usable combination of Wei et al.'s metacognitive prompting with pragmatic categories. The paper includes a worked example, which helps. If you want to build a better sarcasm prompt, this is a reasonable starting point.\n\nThe soft spots are load-bearing for the empirical claim. There are no error bars, no multiple runs, no significance tests. The ToC row for LLaMA-3-8B appears without any training details, despite ToC requiring explicit training. The MUStARD citation points to a name-tagging paper, and the figure references are broken. The abstract overstates: PMP is not uniformly SOTA; zero-shot Claude beats it on MUStARD.\n\nThe core idea is not broken. This is a prompt-engineering paper, and the contribution is the prompt. But the results need a clean rerun with honest baselines, repeated runs, and confidence intervals before anyone should quote the numbers. I would not cite the numbers as they stand.\n\nWho is this for? Researchers working on prompting for figurative language or on why LLMs miss sarcasm. It deserves a serious referee, but the referee should send it back for major revision — fix the baseline comparison, rerun everything, and tighten the claims. If the GPT-4o gap survives a clean rerun, that is a useful result. If not, it is a cautionary tale about baseline hygiene.","headline":"A plausible prompt recipe whose SOTA claim is undercut by copy-pasted baseline rows and no statistical support.","tokens_in":9619,"tokens_out":3119,"would_cite":false,"duration_ms":28053,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pragmatic Metacognitive Prompting achieves state-of-the-art sarcasm detection scores with GPT-4o by making the model analyze implicature, presupposition, intent, polarity, pretense, and meaning before reflecting on its own answer.","keywords":["sarcasm detection","large language models","prompting","pragmatics","metacognition","irony detection","MUStARD","SemEval 2018 Task 3"],"falsifier":"Run PMP and all baseline prompts (zero-shot, CoT, ToT, BoC, CoC, GoC, ToC) on the same GPT-4o checkpoint, temperature, sampling seed, and evaluation script for both benchmarks, with at least three repetitions; if PMP does not beat the best reproduced baseline on both datasets, the paper's central SoTA claim fails. A secondary check is to audit Table 1's baseline rows for transcription errors, since identical numbers appear for different models.","tokens_in":8557,"feed_emoji":"😏","tokens_out":11326,"duration_ms":95014,"temperature":0.7,"pith_summary":"The paper proposes Pragmatic Metacognitive Prompting (PMP), a two-pass prompting scheme in which an LLM first analyzes a statement through six pragmatic lenses—implicature, presuppositions, speaker intent, polarity, pretense, and the gap between literal and implied meaning—and then reflects on that preliminary analysis before issuing a sarcasm verdict. The authors test PMP with GPT-4o, GPT-4o-mini, LLaMA-3-8B, LLaMA-3-70B, and Claude 3.5 Sonnet on the MUStARD and SemEval 2018 Task 3 benchmarks. Their central claim is that PMP pushes GPT-4o to the best sarcasm-detection scores among the prompting methods they compare, including the cue-based SarcasmCue approach, on both datasets. If correct, the result matters because it suggests that importing pragmatic theories of how humans detect irony into prompts, plus a reflection step, can narrow the gap between general-purpose LLMs and specialized sarcasm models.","feed_headline":"Two-pass pragmatics prompt lifts GPT-4o sarcasm scores","feed_subtitle":"Asking the model to judge implicature, pretense and intent before reflecting beats cue-based prompts on both benchmarks.","key_machinery":"The load-bearing mechanism is the two-stage PMP prompt. In the first LLM call, the model is told to summarize the dialogue, repeat the bracketed target statement, and write a preliminary analysis covering six pragmatic factors: implicature (what is implied beyond the literal words), presuppositions (what is taken for granted), the speaker's intent, the polarity or tone of the final sentence, whether the speaker is pretending an attitude, and the difference between literal and implied meaning. In the second LLM call, that preliminary analysis is handed back with instructions to reflect on it, revise it, and produce a final sarcastic or non-sarcastic verdict. This is Metacognitive Prompting—repeat, preliminary judgment, reassessment, final answer—augmented with pragmatic theory; the prompt itself, not any fine-tuning, carries the claimed gain.","core_discovery":"On the paper's own terms, the discovery is that prompting an LLM to reason like a pragmaticist—explicitly asking it to identify implicatures, presuppositions, speaker intent, polarity, pretense, and the divergence between literal and implied meaning, and then to critique its own first-pass answer—materially improves sarcasm detection. The headline evidence is that GPT-4o with PMP reaches 86.68 accuracy and 83.18 macro-F1 on SemEval 2018 Task 3 and 79.42 accuracy and 77.65 macro-F1 on MUStARD, surpassing every prompting baseline listed in the paper's Table 1, including SarcasmCue's Graph of Cues (74.03/74.02 on SemEval). The paper is careful to phrase the SoTA result as 'on GPT-4o,' because the improvement does not hold uniformly across models: on some benchmarks Claude 3.5 Sonnet responds better to zero-shot prompting than to PMP.","pith_inferences":["An ablation that replaces the six pragmatic questions with six generic reflection questions on the same two-pass structure would separate the contribution of pragmatics from the contribution of extra inference compute; the paper does not run this control.","Because PMP first asks the model to name presuppositions and implicatures in its own words, its accuracy is bounded by the LLM's pretrained knowledge of cultural and linguistic norms; the authors acknowledge that underrepresented norms are not handled, so gains may be uneven across dialects and languages.","The same two-call pragmatic template could be tested on adjacent phenomena—indirect requests, deception, hyperbole—where the six dimensions, especially pretense and implicature, are also thought to be central.","A randomized rerun of the baselines under identical sampling conditions is the natural next check; the reported Table 1 contains identical numbers for different models, so some baseline rows may be transcription artifacts rather than genuine measurements."],"forward_implications":["GPT-4o with PMP outperforms zero-shot, Chain-of-Thought, Tree-of-Thought, and all tested SarcasmCue variants on both MUStARD and SemEval 2018 Task 3, on the paper's reported numbers.","PMP also improves LLaMA-3-8B on SemEval 2018, and LLaMA-3-70B outperforms the 8B version, suggesting the benefit scales with model capability.","The method is compute-heavy relative to single-pass prompts because it always spends two LLM calls per example, so part of its advantage may come from extra inference steps rather than pragmatic content alone.","The two-call design produces a transparent reasoning trace that a human can inspect to see why the model judged a statement sarcastic, as shown in the paper's snowy-commute example."],"supporting_citations":[{"why":"Defines the SarcasmCue prompting baselines that PMP claims to surpass.","marker":"(Yao et al., 2024)"},{"why":"Supplies the metacognitive prompting template that PMP extends.","marker":"(Wei et al., 2023)"},{"why":"Provides the MUStARD benchmark used in evaluation.","marker":"(Castro et al., 2019)"},{"why":"Provides the SemEval 2018 Task 3 benchmark used in evaluation.","marker":"(Van Hee et al., 2018)"},{"why":"Provides the pretense theory of irony that PMP turns into a prompt question.","marker":"(Clark and Gerrig, 1984)"},{"why":"Provides the echoic reminder theory of verbal irony that PMP turns into a prompt question.","marker":"(Kreuz and Glucksberg, 1989)"},{"why":"Motivates the reflection stage in PMP's second LLM call.","marker":"(Shinn et al., 2023)"},{"why":"Provides the Chain-of-Thought prompting baseline PMP is compared against.","marker":"(Wei et al., 2022)"},{"why":"Provides the Tree-of-Thoughts prompting baseline PMP is compared against.","marker":"(Yao et al., 2023)"}],"fun_headline_variants":["Pragmatic prompts sharpen GPT-4o's sarcasm radar","Metacognitive prompting boosts GPT-4o sarcasm detection","Ask GPT-4o to think pragmatically, detect sarcasm better","PMP prompt adds pragmatics to boost GPT-4o sarcasm","Reflective prompting helps LLMs flag sarcasm, GPT-4o leads"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim depends on the baseline numbers being trustworthy and comparable: PMP is compared against SarcasmCue results taken from another paper rather than rerun under identical conditions, and Table 1 contains duplicate-looking rows across models, so if those numbers are wrong or not comparable, PMP's state-of-the-art claim is unsupported.","fun_headline_variants_meta":{"raw":{"variants":["Pragmatic prompts sharpen GPT-4o's sarcasm radar","Metacognitive prompting boosts GPT-4o sarcasm detection","Ask GPT-4o to think pragmatically, detect sarcasm better","PMP prompt adds pragmatics to boost GPT-4o sarcasm","Reflective prompting helps LLMs flag sarcasm, GPT-4o leads"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000484,"raw_usage":{"total_tokens":2368,"prompt_tokens":904,"completion_tokens":1464,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":1370}},"tokens_in":520,"tokens_out":1464,"duration_ms":10661,"temperature":1.0,"reasoning_tokens":1370,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:47:03.454750+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run PMP and all baseline prompts (zero-shot, CoT, ToT, BoC, CoC, GoC, ToC) on the same GPT-4o checkpoint, temperature, sampling seed, and evaluation script for both benchmarks, with at least three repetitions; if PMP does not beat the best reproduced baseline on both datasets, the paper's central SoTA claim fails. A secondary check is to audit Table 1's baseline rows for transcription errors, since identical numbers appear for different models.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MUStARD benchmark used in evaluation."}],"review_version":1}