{"id":"c690e64f-9364-459f-8f42-a2c764ba1177","arxiv_id":"2602.06423","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A GTVH-driven multi-role LLM framework with conflicting-script extraction and humor-retrieval imagination trees improves cartoon caption funniness over previous methods on New Yorker benchmarks.","lead":"This paper introduces HOMER, a three-role LLM pipeline that uses script-opposition theory, imagination trees, and joke retrieval to write funny captions for cartoons. It reports consistent wins over seven baselines on two New Yorker caption benchmarks, though the evaluation depends heavily on GPT-5 as judge.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed superiority rests on GPT-5 as humor judge, validated only on human-written pairs; the human backstop is internally inconsistent (20 vs. 12 raters; 5,600 vs. 2,800 ratings), leaving the core result conditional on judge transfer.","rationale":"The reader's weakest_assumption—that GPT-5-as-judge is not sufficiently validated on generated captions and the human evaluation is internally inconsistent—is exactly the load-bearing concern. The central claim is an empirical superiority claim; its truth depends on the evaluation metric faithfully reflecting human humor preferences. The paper's own validation of GPT-5 is limited to 200 human-written pairs, and the human evaluation that could backstop it is small and contradictory in reported rater counts and scores. I agree with the reader's CONDITIONAL verdict: the method is coherent and well-ablated, with equal-call comparisons and a leakage analysis, but the core result is conditional on independent human validation of the generated captions. I am not moving the verdict because the concern is the same one the reader already identified; no new fatal flaw emerged. The concrete test I propose would settle whether the concern lands by directly measuring human preferences on the actual system outputs.","tokens_in":28862,"tokens_out":6125,"duration_ms":67550,"concrete_test":"Pre-register a human preference study on the actual generated outputs: sample 100 cartoons from Humor in AI and 100 from Electronic Sheep, collect 5 captions each from HOMER and the three strongest baselines (HumorousAI, CLoT, LoL) using the exact Table 3 settings, and have 30 independent raters perform forced-choice pairwise comparisons (HOMER vs. each baseline) and 1–5 funniness ratings. Compare the human win rates and margins to the GPT-5-derived pass@k differences. If HOMER's margin over the best baseline is smaller than the GPT-5 margin, or not statistically significant, the central claim is not supported. This directly tests judge transfer and sidesteps the inconsistent rater counts in the paper.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—HOMER outperforms seven baselines by 8.62% (pass@1), 6.48% (pass@3), and 5.91% (pass@5) on two New Yorker benchmarks—rests entirely on pass@k scores computed by GPT-5 as a stand-in for human humor preference. Table 2 validates GPT-5 only on 200 human-written caption pairs (73.5% ranking accuracy). The paper then uses this judge to compare machine-generated captions against human captions, a different distribution with no direct validation. If GPT-5 systematically favors the explicit script-opposition/imagination-tree phrasing that HOMER produces, the reported margins could be an artifact of judge bias rather than genuine humor gains. The human evaluation intended as a backstop is too small and internally inconsistent to resolve this: the main text reports 20 raters and 5,600 ratings, Appendix B.8 reports 2,800 ratings (20 images × 20 raters × 7 methods), and the 'Detailed human evaluation' says 12 raters with different means (3.51/3.38 vs. 3.54/3.31 in Table 6). No released outputs or artifacts are provided to independently audit the judge's behavior on generated captions. Appendix B.14's GPT-4.1 corroboration does not settle the issue because GPT-4.1 is also an LLM judge with only 68.5% accuracy and moderate human correlation (0.56). Thus the load-bearing assumption—that GPT-5's judgments transfer to generated captions—is unverified, and the only direct human evidence is unreliable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HOMER, a three-role LLM framework for multimodal humor caption generation. Grounded in the General Theory of Verbal Humor (GTVH), HOMER first extracts situation descriptions and conflicting scripts from an image, then builds hierarchical 'imagination trees' through LLM free association and retrieval from a joke database with a humor-relevance pruning score, and finally generates captions conditioned on the selected scripts, targets, and imagination paths. The method is evaluated on the Humor in AI and Electronic Sheep New Yorker Cartoon benchmarks, with GPT-5 as the primary humor judge for pass@k metrics, plus human ratings, ablations, a meme generalization experiment, and toxicity checks. The paper claims average improvements of 8.62%, 6.48%, and 5.91% over strong baselines on pass@1, pass@3, and pass@5, respectively, with GPT-4o as the base generator.","tokens_in":29293,"tokens_out":6770,"duration_ms":76943,"significance":"If the headline results are accepted, HOMER would be a meaningful advance in multi-modal humor generation: it is one of the few frameworks that operationalizes a specific humor theory (GTVH) in an interpretable pipeline, and its design choices are tested unusually thoroughly. Strengths include exhaustive module ablations (Table 4), hyperparameter sensitivity analyses (Figures 4-5), evaluation across four base LLMs (Table 3), a cross-domain meme experiment (Table 5), a leakage analysis (Appendix B.15), and significance tests (Appendix B.13). The paper also is candid about failure cases in Appendix D. The central reservation is that the headline pass@k numbers rest on GPT-5 as a stand-in for human humor judgment, validated only on human-written caption pairs, while the direct human evaluation is small and internally inconsistent; the load-bearing evaluator-transfer assumption is therefore not yet established.","major_comments":[{"comment":"The human evaluation numbers are irreconcilable as reported. Section 3.5 states 20 raters and 5,600 rating scores. Appendix B.8 first describes 2,800 data points (20 images × 20 raters × 7 methods), then the 'Detailed human evaluation' paragraph says 12 raters and reports HOMER means of 3.51 (Humor in AI) and 3.38 (Electronic Sheep), while Table 6 and Table 13 report 20 raters and means of 3.54±0.59 and 3.31±0.85. These are different experiments or errors; as written, the only direct human backstop for the GPT-5 judge cannot be used to verify the claimed margins. This must be corrected and the actual protocol, rater count, and per-method scores reported consistently.","section":"Section 3.5 and Appendix B.8"},{"comment":"GPT-5 is validated only on 200 human-written caption pairs (73.5% and 70.0% ranking accuracy in Table 2), but the main pass@k results in Table 3 use GPT-5 to compare model-generated captions against human-written captions. No validation on the generated-caption distribution is provided. The GPT-4.1 corroboration (68.5% accuracy, Pearson 0.5639 against human ratings) is another LLM judge and does not resolve the transfer concern. If GPT-5 systematically prefers HOMER's explicit script-opposition style, the reported improvements could be an artifact. The paper should either report judge accuracy/correlation on a sample of generated captions labeled by humans, or release the generated outputs so the judge's behavior can be independently audited.","section":"Section 3.1, Section 3.2, Appendix B.14"},{"comment":"The headline pass@k results are means over five runs with no standard deviations, confidence intervals, or per-run distributions. Generation uses temperature 1.0 and random selection of scripts/paths, so run-to-run variability is expected. Several margins over the strongest baseline are small (e.g., Claude-4 rows: +1.25 and +1.26 on #200-209 groups), and the reported Wilcoxon tests are pairwise tests on aggregate pass@k values without correction. Please report the five per-run scores (or confidence intervals) and make the significance testing procedure explicit; otherwise the 1-3 point margins are hard to distinguish from sampling noise.","section":"Table 3 and Appendix B.13"}],"minor_comments":[{"comment":"The 'Avg #captions' entry for Humor in AI is shown as '6,044', which is implausible as an average per cartoon. Please clarify whether this is a total, a misprint, or a different statistic.","section":"Table 1"},{"comment":"The notation for n_i and c_i should be tightened. The main text leaves n_i implicit until Appendix B.3 (where it is set to 5); state this in the main text and explain the interpretation of pass@5 when n_i=5, since pass@5 then reduces to the empirical win-rate over the five generated captions.","section":"Eq. (7) and Appendix B.3"},{"comment":"The text after Eq. (4) promises a 'detailed proof of convergence and monotonicity' in Appendix C, but Appendix C contains only informal boundedness and monotonicity remarks; no formal proof or theorem statement is given. Either supply a precise statement/proof or soften the claim.","section":"Section 2.2 and Appendix C"},{"comment":"The text reports the summed toxicity as 0.023 and 0.015, while Figure 7's axis is labeled 'Sum of toxicity (%)' with values 2.35 and 1.52. Make the units consistent (proportion vs. percentage) so the numbers can be compared directly.","section":"Section 3.6 and Figure 7"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-engineered and the ablation design is a model of thoroughness, but the headline empirical claim is currently conditioned on an unvalidated GPT-5 judge transfer, and the human evaluation meant to backstop it contains contradictory counts and scores. I would need to see a corrected, internally consistent human evaluation and either a judge-transfer validation on generated captions or released outputs for independent audit before recommending acceptance. The framework itself is defensible and the central idea is strong; the issues are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the framework is real: GTVH script-opposition extraction, hierarchical imagination trees, and humor-relevance retrieval are integrated into a coherent pipeline, and the ablations actually support each piece. Second, the headline numbers should be read as conditional. All pass@k margins are computed by GPT-5 as the humor judge, validated only on 200 human-written pairs (73.5% accuracy). The human evaluation meant to backstop this is internally inconsistent—20 raters and 5,600 ratings in the main text, 2,800 in Appendix B.8, and 12 raters in the detailed human evaluation, with different means (3.51/3.38 vs 3.54/3.31). No generated captions or artifacts are released, so you cannot audit the judge on the actual distribution.\n\nWhat is genuinely good: the design is interpretable and modular. The conflicting-script extractor gives a traceable reason for each caption; the imagination tree with retrieval is a reasonable way to inject creativity; the HOMER-pruning score is a bit ad hoc, but the ablation shows each term matters. Hyperparameters k and delta are explored, and the method is stable across the tested range. Diversity metrics and toxicity checks are included. The GPT-4.1 corroboration exists, though it is also an LLM judge with 68.5% accuracy and moderate human correlation.\n\nWeak spots, in proportion: the evaluation chain is the load-bearing issue. If GPT-5 systematically prefers HOMER's explicit script-opposition phrasing, the reported 8.62% pass@1 improvement could be an artifact. That is not a manufactured concern; the paper does not demonstrate judge transfer to generated captions. The human evaluation inconsistencies are real and need fixing, not just cosmetic. Pass@k is reported without variance, and the significance tests run on GPT-5 scores inherit any judge bias. The theoretical analysis of H_rel in Appendix C is fine but not deep—boundedness and monotonicity, which is enough for this purpose.\n\nWho this is for: anyone working on humor generation or creative captioning. The framework is worth citing and building on, even if the empirical margins need independent verification. I would send it to a serious referee before trusting the numbers; the design merits review, and the evaluation flaws are fixable. Recommendation: engage with the work, but ask for released outputs and a clean human evaluation on generated captions.","headline":"HOMER is a solid, interpretable framework for humor captioning, but the headline win rates are only as trustworthy as GPT-5's transfer to generated captions, and the human backstop is inconsistent.","tokens_in":29706,"tokens_out":3105,"would_cite":true,"duration_ms":29983,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HOMER claims that funnier cartoon captions come from extracting conflicting scripts and building imaginative association trees before the caption is written, and that this outperforms reasoning-only baselines.","keywords":["humor generation","multimodal captioning","script opposition","GTVH","large language models","imagination trees","retrieval-augmented generation","cartoon captions"],"falsifier":"Collect pairwise human funniness judgments on a random sample of, say, 100 HOMER captions versus the strongest baseline captions on the same images, using multiple raters; if the human win-rate margin is much smaller than the LLM-judge pass@k margin, or reverses, the central claim fails.","tokens_in":28793,"feed_emoji":"😂","tokens_out":5991,"duration_ms":67898,"temperature":0.7,"pith_summary":"The paper tries to show that large language models generate funnier cartoon captions when generation is explicitly organized around a humor theory rather than left to open-ended reasoning. It introduces HOMER, which first extracts the situation and the conflicting scripts that make an image funny, then builds imagination trees linking humor targets to surprising associations via LLM free association and joke retrieval, and finally generates a caption from that material. Across two public cartoon-caption benchmarks, HOMER reports average gains of roughly 8.6% at pass@1, 6.5% at pass@3, and 5.9% at pass@5 over seven baselines with a strong commercial base model, with consistent gains on other base models and a meme dataset. A sympathetic reader would care because the intermediate steps are interpretable: you can see which script opposition and which associative path produced a joke, something chain-of-thought approaches do not offer.","feed_headline":"Conflicting-script pipeline lifts LLM humor captions by 8.6%","feed_subtitle":"A three-role pipeline—conflicting scripts, imagination trees, generation—beats seven baselines on cartoon captions.","key_machinery":"The central machinery is the General Theory of Verbal Humor applied as script opposition—the idea that humor arises when two conflicting semantic frames are set against each other—plus a hierarchical imagination tree. The tree is built in two passes: deep chains from LLM free association form the backbone, and retrieval from a joke database adds leaves, with a humor-relevance score combining semantic similarity, conceptual opposition, humor frequency, and part-of-speech diversity to prune weak branches. This machinery carries the creative search: it converts a static image into a structured space of possible joke paths that the generator can sample.","core_discovery":"On the paper's own terms, the central discovery is that script opposition can serve as a computational scaffold for humor generation. HOMER divides caption writing into three roles: an extractor that names the situation and conflicting scripts; an imaginator that grows each humor target into an imagination tree through LLM associations and joke-corpus retrieval, pruned by a humor-relevance score; and a generator that samples a script, a target, and an associative path. The paper reports that this structure outperforms seven state-of-the-art baselines on two cartoon-caption benchmarks, that removing any one module degrades performance, and that the largest drop occurs when conflicting scripts","pith_inferences":["The explicit conflicting-script representation may transfer to other constrained creative-generation tasks—puns, headlines, ad copy—where a surprising juxtaposition of two frames is the core move.","The humor-relevance pruning score could be reused as a cheap filter for joke retrieval in dialogue or recommendation systems, independent of HOMER.","A testable extension: compare HOMER against a version given the same script-opposition annotations but forced to generate linearly, to isolate whether the hierarchical tree or the script signal drives the gains.","If future human studies reproduce the reported margins, this would support the stronger claim that LLMs' humor weakness is a problem of structured imagination, not of reasoning depth."],"forward_implications":["If the claim holds, LLM cartoon captioning improves measurably just by adding theory-structured intermediate steps, without task-specific fine-tuning.","The two public benchmarks' pass@k win rates over human captions rise with HOMER, not only with strong proprietary base models but also with smaller open models.","The interpretable intermediate outputs—conflicting scripts and sampled imagination paths—allow a user to see and edit the joke logic before the caption is fixed.","Each component (situation description, conflicting scripts, imagination trees) is reported necessary; removing scripts causes the largest performance drop.","Generalization results on a meme dataset suggest the same conflicting-script recipe transfers beyond the benchmark's cartoon style."],"fun_headline_variants":["Conflicting scripts make LLMs funnier captioners","HOMER's script-clash trio outwrites seven rivals","Imagination trees plus script opposites lift humor","GTVH-inspired multi-role framework boosts caption wit","Two-script humor extraction sharpens LLM captions"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that a large language model humor judge, validated on 200 human-ranked caption pairs at about 73.5% accuracy, transfers that accuracy to ranking generated captions from a different distribution; if it does not, HOMER's reported margins may reflect judge preference rather than real funniness.","fun_headline_variants_meta":{"raw":{"variants":["Conflicting scripts make LLMs funnier captioners","HOMER's script-clash trio outwrites seven rivals","Imagination trees plus script opposites lift humor","GTVH-inspired multi-role framework boosts caption wit","Two-script humor extraction sharpens LLM captions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000177,"raw_usage":{"total_tokens":1142,"prompt_tokens":766,"completion_tokens":376,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":297}},"tokens_in":510,"tokens_out":376,"duration_ms":4725,"temperature":1.0,"reasoning_tokens":297,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T06:06:32.457940+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect pairwise human funniness judgments on a random sample of, say, 100 HOMER captions versus the strongest baseline captions on the same images, using multiple raters; if the human win-rate margin is much smaller than the LLM-judge pass@k margin, or reverses, the central claim fails.","supporting_citations":[],"review_version":1}