{"id":"6457ec1b-f705-4884-856b-deb35fa32e8e","arxiv_id":"2412.05315","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Adding TTS audio of jokes to text prompts improved LLM humor explanation win rates by roughly 3 to 4 percentage points across pun and joke datasets.","lead":"This paper feeds both the text and a text-to-speech audio version of a joke to Gemini-1.5-Flash and asks it to explain the humor, comparing the result with text-only prompts. The added audio raises explanation win rates by roughly three to four percentage points across three datasets, suggesting that LLMs can use phonetic cues when they are made audible.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains of ~2–4 points rest entirely on GPT-4o pairwise preference without human validation or significance tests; the effect is within the noise band for SemEval sample sizes, so the central claim is not yet established.","rationale":"The paper attempts a clean, practical intervention: add TTS audio to text prompts and show that LLM humor explanations improve. For the central claim to hold, two things must be true: the audio must carry useful phonetic information, and the evaluation must measure explanation quality accurately enough to detect a real difference. The first is plausibly supported by the logit analysis in Appendix A.2, which shows the open-source transcriber assigns high probability to both spellings of the pun word, and by the ablation showing audio-only prompting is much worse than text-only. The second is the weak link. The only human-anchored comparison is on SemEval, and even there the judge is GPT-4o rather than a human rater; the other two datasets compare the model against itself via the same LLM judge. The win-rate gaps are 2–4 points, and with SemEval sample sizes of roughly 650–810 items, a paired difference of that size is within the noise band (approximate 95% CI crossing zero). The paper's own Limitations section acknowledges that automated judging may not capture explanation quality, which strengthens the concern that the headline effect is not yet established. I agree with the reader's identification of this as the weakest assumption. A paired bootstrap on the existing judgments would settle the significance question, and a modest human preference study would settle the validity question. If both pass, the paper's claim is substantially supported; if either fails, the reported improvements may be an artifact of the evaluation setup rather than genuine humor understanding. The verdict should remain conditional because the issues are concrete and addressable, not fatal to the approach.","tokens_in":8404,"tokens_out":7411,"duration_ms":74255,"concrete_test":"Re-analyze the raw per-item GPT-4o judgments from Tables 1–3 with a paired bootstrap: resample jokes with replacement, keep the baseline/audio/tie decision for each joke, and compute 95% confidence intervals for each win-rate difference; if any interval includes 0, the reported gains are not statistically reliable. Then recruit three human annotators to judge a random subset of about 100 SemEval pairs using the same annotation-informed rubric, and compare human preferences with GPT-4o's; if human ratings do not show a comparable audio-over-baseline advantage, or if annotator agreement with GPT-4o is low, the LLM-judge proxy rather than humor understanding is driving the reported result.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's headline claim ('Using multimodal cues improves the explanations of humor...') is supported by win-rate differences of 3–4 points on SemEval (Table 1: heterographs 47.76→51.74; homographs 68.89→72.59) and ~2–3 points on Context-Situated Puns and ExplainTheJoke (Tables 2–3). All comparisons are judged by GPT-4o pairwise preference (Section 4.2; Appendix A.1.2), not by humans; no confidence intervals or significance tests are reported. The judge is given the pun annotations, which makes the rubric more objective, but GPT-4o remains a stochastic preference model with known stylistic and positional biases, and the generation prompt for the audio condition explicitly tells the model to consider the audio (Figure 5), so the two explanation sets can differ in framing or length ('hearing it aloud helps...'). Section 7 concedes that LLM-based judging 'may not fully capture the nuanced quality of humor explanations.' For the SemEval sample sizes (roughly 647–810 puns), a 3–4 point difference is within the approximate ±3–5 point 95% noise band, so even the judge's own preferences do not establish the effect. The load-bearing assumption is that GPT-4o's pairwise preference tracks human explanation quality; this is cited from prior work (Zheng et al., 2024) but never validated on this task.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multimodal prompting approach for LLM humor explanation: in addition to the joke text, the model receives an audio rendering generated by an off-the-shelf TTS system (OpenAI tts-1-hd). The method is evaluated on three datasets—SemEval 2017 Task 7, Context-Situated Puns, and ExplainTheJoke—using GPT-4o as an LLM judge to compare explanations from text-only prompts against those from text+audio prompts. The paper reports consistent win-rate improvements of roughly 2–4 percentage points across all datasets, an ablation study, a voice-sensitivity analysis, and a small logit-based analysis of phonetic ambiguity in TTS audio.","tokens_in":8689,"tokens_out":4722,"duration_ms":42686,"significance":"If the reported improvements are real, the paper offers a simple, training-free method that could improve humor understanding in LLMs and serve as a building block for multimodal reasoning. The approach is reproducible : it uses off-the-shelf TTS, publicly available datasets, and fully specified prompts in the appendix. The paper also makes a useful mechanistic observation in Appendix A.2 that TTS audio preserves homophone ambiguity, and it reports an ablation showing that audio-only input degrades performance. However, the significance is currently undermined by the evaluation methodology: the central claim rests on small win-rate differences that are not accompanied by significance tests or confidence intervals, and two of the three datasets lack any human baseline or human validation of the LLM judge.","major_comments":[{"comment":"The central claim that multimodal prompting 'significantly improves' humor explanations (Section 6) is not supported by the reported statistics. The win-rate differences are small: for SemEval heterographs, 47.76% to 51.74% (n=647); for homographs, 68.89% to 72.59% (n=810); for Context-Situated Puns, 33.87% to 36.49% (heterographs) and 35.08% to 36.85% (homographs); and for ExplainTheJoke, 12.81% to 15.44% (n=350). No confidence intervals, p-values, or effect sizes are reported anywhere in the manuscript. Assuming independent two-proportion comparisons, these differences are within the approximate 95% sampling error band (e.g., the heterograph difference of 3.98 points has a standard error of about 2.8 points). The authors should report paired analyses (e.g., McNemar's test if each joke is evaluated under both conditions) or at least confidence intervals, and should qualify the abstract and conclusion accordingly if the differences are not statistically significant.","section":"Section 5, Tables 1–3"},{"comment":"The evaluation relies entirely on GPT-4o pairwise preference as a proxy for explanation quality. For Context-Situated Puns and ExplainTheJoke, there is no human-written reference: the judge compares two LLM-generated explanations (baseline vs. audio-conditioned). This makes the quality signal partly self-referential, especially because the generation prompt (Figure 5) explicitly instructs the model to consider the audio and to check if hearing it aloud helps, which can systematically alter response length, framing, or content in ways that a stylistic-preference judge might reward regardless of actual humor understanding. The paper cites Zheng et al. (2024) for the validity of LLM judges, but that work addresses general chat response quality, not humor explanation quality. The authors should validate the judge against human ratings on a sample of this task, or at minimum report agreement between GPT-4o and human annotators on the SemEval subset where human explanations exist.","section":"Section 4.2 and Section 5.2–5.3"},{"comment":"The ablation results indicate that the multimodal gain is highly sensitive to prompt wording. Removing the instruction to avoid discussing the input modality reduces the heterograph win rate from 51.74% to 48.61%, which is only 0.85 points above the text-only baseline of 47.76%. The audio-only condition performs far worse (25.50% heterographs, 55.86% homographs). This pattern suggests that a substantial part of the reported improvement may come from the added instruction or prompt complexity rather than from the acoustic information itself. The paper should control for this confound, for example by adding an equivalent 'read aloud' instruction to the text-only condition or by measuring whether the audio condition produces longer or differently structured explanations, and then analyzing whether those features account for the judge's preference.","section":"Appendix A.3, Table 4"},{"comment":"The logit analysis intended to demonstrate that audio preserves phonetic ambiguity is anecdotal and uses a different model (Gazelle v0.2) than the generation model used in the main experiments (Gemini-1.5-Flash). It is based on a single word ('Where') and a single pun ('Patience is a heavy weight'), so it does not establish that the TTS audio in the actual test set preserves the relevant homophone distinctions for the model being evaluated. The authors should either run this analysis systematically across a sample of the test puns using the actual model, or clearly label the result as a preliminary illustration and remove the claim that it explains the main results.","section":"Appendix A.2, Figures 3–4"}],"minor_comments":[{"comment":"Table 2 omits tie percentages for the 'with audio' rows, and Table 3 omits the tie percentage for the 'with audio' row, making it impossible to infer the full ternary distribution. Please report tie rates for all conditions.","section":"Tables 2 and 3"},{"comment":"The text says performance increased 'by approximately 4%' in both cases, but the exact increases are 3.98 and 3.70 percentage points. Use exact figures or percentage-point terminology to avoid ambiguity.","section":"Section 5.1"},{"comment":"The reference 'Chris Hua. 2024. Gazelle v0.2' lacks a venue, version, or access URL; if this is a model or repository, please provide a citation or a footnote description.","section":"References"},{"comment":"The Ethics Statement begins with 'Large Large Language Models', which appears to be a typo for 'Large Language Models'.","section":"Section 8"},{"comment":"The claim that including few-shot examples 'ensured a consistent output tone' is plausible but not verified; please either provide evidence (e.g., output length statistics) or soften the claim.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is clearly written and addresses an interesting problem, but the central claim is currently not established because the reported effect sizes are small, no significance testing is provided, and the evaluation is entirely LLM-based for two of three datasets. The requested revisions—adding significance tests, human validation of the judge, and control for prompt confounds—are within the scope of a journal-length revision. If the authors cannot add human evaluation, the paper might be better suited to a workshop-style venue. I do not see fundamental flaws in the approach itself; the concern is purely about evidence strength."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this paper tries a cheap trick—feed the LLM the text of a joke plus an off-the-shelf TTS audio clip—and reports that it improves humor explanations by a few points across three datasets. The idea is genuinely new in this form and the write-up is clear, but the evidence for the headline claim is weaker than the abstract suggests.\n\nWhat is actually good: the method is simple, reproducible, and builds directly on prior text-only work (Xu et al.). The ablations are useful: audio-only prompting is much worse than text-only, the wording that tells the model not to mention modalities matters a bit, and voice choice makes no consistent difference. The logit analysis of transcription, showing \"weight\" and \"wait\" both get high probability, is a nice mechanistic sanity check that the audio preserves phonetic ambiguity.\n\nThe soft spot is the evaluation. All win rates come from GPT-4o as judge. For two of the three datasets there is no human benchmark at all, only LLM-vs-LLM preference. The differences are 2–4 points, which is within the noise band for the SemEval sample sizes: for 647 heterographs, a 3.98-point gap is about 1.4 standard errors. The paper also selects the final prompt configuration after seeing test-set results, and the generation prompt explicitly tells the model to consider the audio, so the two explanation sets may differ in style as much as in content. The paper concedes in Section 7 that LLM judging \"may not fully capture the nuanced quality of humor explanations,\" but the abstract still says \"significantly improves.\"\n\nI would not go so far as to call the claim false. The consistency across all five comparisons is suggestive, and the effect, if real, would be useful for speech-based assistants. But the central result is not established with the current evidence. A human-rated subset of a few hundred explanations and confidence intervals would settle it.\n\nWho is this for? People working on multimodal prompting or humor understanding. It deserves a serious referee, but the referee should ask for stronger evaluation before publication.\n\nMy recommendation: send it to review, but brace the authors for the request to add human judgments and significance tests. It is a solid workshop-to-conference quality idea that needs one more round of evidence.","headline":"A cheap and plausible trick—adding TTS audio to prompts—yields small, consistent win rates in LLM humor explanation, but the LLM-as-judge evaluation is too weak to establish the claim.","tokens_in":9219,"tokens_out":2567,"would_cite":false,"duration_ms":23846,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that adding synthesized speech to an LLM's prompt improves its humor explanations over text-only prompting.","keywords":["multimodal prompting","humor understanding","pun explanation","text-to-speech","phonetic ambiguity","LLM-as-judge","chain-of-thought","few-shot prompting"],"falsifier":"Run the same pairwise comparisons with human raters on a sample of the SemEval outputs, or bootstrap the win rates across the dataset; if the multimodal advantage disappears or falls within sampling noise, the central claim is not supported.","tokens_in":8192,"feed_emoji":"😂","tokens_out":7196,"duration_ms":57760,"temperature":0.7,"pith_summary":"This paper claims that supplying an LLM with the spoken form of a joke, synthesized by an off-the-shelf text-to-speech system, improves its written explanations of the humor compared with giving it only the text. On the SemEval 2017 pun dataset, the multimodal prompt raised the model's win rate against human-written explanations from 47.76% to 51.74% for heterographs and from 68.89% to 72.59% for homographs. The same pattern appears on two other joke datasets when multimodal outputs are compared directly with text-only outputs. The author argues that jokes, and puns in particular, depend on phonetic ambiguity that token-based text encoding flattens away, while the audio channel preserves it. The approach is training-free and applicable to any LLM that accepts audio input.","feed_headline":"Adding audio improves LLM humor explanations","feed_subtitle":"TTS-spoken jokes beat text-only prompts by ~4 points on SemEval pun explanations.","key_machinery":"The load-bearing mechanism is a multimodal prompt that concatenates the joke's text with an audio clip generated by an off-the-shelf text-to-speech model, sent together to an LLM that accepts audio input. Because heterographs are distinct tokens in text but identical in pronunciation, the speech track carries the ambiguity that the text-only prompt loses; the prompt instructs the model to reason from both channels and to produce an explanation that does not mention the modalities. Evaluation runs through a separate LLM judge that performs pairwise comparisons with swapped order to cancel positional bias. Two auxiliary checks complete the mechanism: an ablation shows audio-only prompting is far worse than text-only, and transcription logits show both homophones of a pun word receive high probability, indicating that the phonetic cue enters the model's representation.","core_discovery":"On the paper's own terms, the central discovery is that adding synthesized speech to the prompt gives LLMs a consistent edge in explaining humor. In the head-to-head comparison against human explanations in SemEval, the multimodal condition is preferred more often than the text-only condition for both homographic and heterographic puns, with win rates rising by about four percentage points in each case. For Context-Situated Puns and ExplainTheJoke, where no human-explanation baseline exists, the author compares the two prompting strategies directly and reports the audio-augmented version winning more often. A logit-based analysis of a transcription task shows that the model assigns high probability to both spellings of the pun word (e.g., \"weight\" and \"wait\"), which the author interprets as evidence that the phonetic ambiguity survives in the model's internal representations. The paper frames this as a demonstration that multimodal prompts are a straightforward way to improve humor understanding without retraining.","pith_inferences":["Because the judge is itself an LLM and the reported gains are a few percentage points, a human-preference study on the same output pairs is the natural next test; if human raters do not reproduce the ordering, the advantage is likely judge style rather than humor understanding.","The transcription-logit result suggests a cheap diagnostic for any text task: if adding a TTS reading changes performance, part of the task's difficulty lives in sound rather than text.","The method could be inverted to test causality: degrading the audio channel—by removing prosody or normalizing pitch—should shrink the gap if the auditory features genuinely carry humor.","Because no confidence intervals or repeated-seed runs are reported, the stability of the roughly four-point effect across samples is untested; bootstrapping the win rate would settle that question."],"forward_implications":["If the central claim holds, any LLM with audio support can improve humor explanation quality by generating speech from the joke text before prompting, with no fine-tuning or dataset collection.","The phonetic-ambiguity mechanism implies that tasks involving homophones—pun detection, transcription, speech disambiguation—are natural beneficiaries of the same multimodal prompting recipe.","The win-rate gains on non-pun jokes (ExplainTheJoke) suggest audio conveys prosodic or rhythmic cues beyond spelling ambiguity, pointing toward a broader role for speech in humor understanding.","The absence of voice-dependent effects in the ablations supports the interpretation that the benefit comes from the audio channel itself rather than from a particular TTS voice.","A direct corollary is that text-only benchmark results for humor explanation may understate what LLMs can do once auditory information is available."],"supporting_citations":[{"why":"Provides the SemEval 2017 Task 7 pun dataset with human explanations that serve as the ground-truth baseline for the main comparison.","marker":"Miller et al., 2017"},{"why":"Supplies the chain-of-thought prompt structure and the guidance to avoid asking for an explanation before detection, which the paper adapts.","marker":"Xu et al., 2024"},{"why":"Justifies LLM-as-judge evaluation and motivates the order-swapping procedure used to mitigate positional bias.","marker":"Zheng et al., 2024"},{"why":"Provides the Context-Situated Puns dataset used for the second benchmark with annotations but no human explanations.","marker":"Sun et al., 2022"},{"why":"Supplies the ExplainTheJoke dataset of jokes with human explanations used as the third benchmark.","marker":"theblackcat102"},{"why":"Documents positional bias in LLM judges, supporting the swapped-order averaging in the evaluation.","marker":"Wang et al., 2024"},{"why":"Establishes few-shot in-context learning, which the explanation prompts rely on for performance and output consistency.","marker":"Brown et al., 2020"}],"fun_headline_variants":["Adding speech to text helps LLMs understand humor","Audio-augmented prompts improve LLM joke explanations","LLMs explain jokes better when they hear them","TTS audio gives LLMs a humor explanation edge"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that an LLM judge's pairwise preference is a valid measure of explanation quality, since no human ratings or significance tests back the roughly four-point gaps.","fun_headline_variants_meta":{"raw":{"variants":["Adding speech to text helps LLMs understand humor","Audio-augmented prompts improve LLM joke explanations","LLMs explain jokes better when they hear them","TTS audio gives LLMs a humor explanation edge"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000271,"raw_usage":{"total_tokens":1564,"prompt_tokens":817,"completion_tokens":747,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":433,"completion_tokens_details":{"reasoning_tokens":686}},"tokens_in":433,"tokens_out":747,"duration_ms":7205,"temperature":1.0,"reasoning_tokens":686,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:04:58.087629+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pairwise comparisons with human raters on a sample of the SemEval outputs, or bootstrap the win rates across the dataset; if the multimodal advantage disappears or falls within sampling noise, the central claim is not supported.","supporting_citations":[],"review_version":1}