{"id":"6b4612a9-b0ee-4364-8c72-e526f83eb709","arxiv_id":"2508.19402","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"LLMs fine-tuned on one to three humor datasets transfer partially to unseen humor types (up to 75% accuracy); diverse training helps modestly, and dad jokes enable transfer best but resist it as a target.","lead":"This paper tests whether an AI trained to recognize one style of humor can recognize other, never-seen styles. Fine-tuning on one to three joke datasets transfers to held-out humor types (up to 75 percent accuracy), with dad jokes the best transfer source but the hardest target.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Target-data leakage in hyperparameter selection (Appendix D) biases the headline transfer and diversity numbers; re-analysis with training-only selection is needed before the central claim can be taken at face value.","rationale":"The reader's weakest assumption was the GPT-4-generated Dad Jokes negatives; that is a real concern for the type-level asymmetry. But the more load-bearing issue is the hyperparameter-selection protocol in Appendix D, which uses held-out target datasets during model selection. This threatens every reported transfer accuracy and the diversity gain, not just Dad Jokes. The paper itself supplies details that make the leakage explicit: 'the configuration with the highest median accuracy across all datasets was selected for final evaluation on test set.' Because the diversity comparison varies both the training data and the number of candidate configurations (3, 6, or 9), the claimed 1.88–4.05 point improvements could partly reflect increased selection over target-validation performance. The paper has real strengths: zero-shot performance is near chance, in-domain accuracy is high, two model families are used, and code/data are released. However, the central quantitative claims cannot be evaluated fairly without re-running with a training-only selection rule. This is a fixable methodological flaw, so the conditional verdict stands, but for a more fundamental reason than the reader's primary concern.","tokens_in":15935,"tokens_out":5641,"duration_ms":65662,"concrete_test":"Re-run the single, double, and triple experiments with hyperparameters selected using only the training dataset's own validation split (or fixed across all setups), never using held-out target data, and recompute Tables 2–4 and Table 7. If the maximum unseen-dataset accuracy drops below ~65% or the average single→triple transfer gain (Table 7: +4.05 for LLaMA-2, +1.88 for Mistral) becomes non-positive for at least one model, the abstract's quantitative claims are not robust. Also report per-seed accuracies so significance can be assessed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline quantities—'up to 75% accuracy on unseen datasets' and 'training on diverse sources improves transferability (1.88–4.05%)'—are produced by a hyperparameter-selection protocol that peeks at the target datasets. Appendix D states that after a 4-fold cross-validation search on each training dataset, the top 3 configurations are chosen 'based on median cross-validation accuracy across all evaluation datasets.' Then, for each experimental setup, 'the configuration with the highest median accuracy across all datasets was selected for final evaluation on test set.' Because 'all datasets' includes the held-out transfer targets, the reported numbers are maxima over configurations selected on those targets, not unbiased estimates under a fixed model-selection rule. The double/triple setups draw from a larger configuration pool (6 or 9 candidates vs. 3), so the measured 1.88–4.05 point diversity gain is confounded with increased selection pressure. This does not prove transfer is absent—cross-dataset accuracies are consistently above chance—but the specific magnitudes, and especially the diversity conclusion in the abstract and Section 5.3, are not supported at face value. The GPT-4-generated Dad Jokes negatives (Section 3.1) are a secondary threat to the type-level asymmetry, but the target-peeking selection affects every reported accuracy and difference.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks whether LLMs can transfer humor-detection competence across four distinct humor datasets (Amazon Questions, Reddit Dad Jokes, Sarcasm Headlines, One Liners). Using LLaMA-2-7B and Mistral-7B, the authors fine-tune on one, two, or three datasets and evaluate on held-out humor types. They report that transfer is possible (up to 75% accuracy on an unseen dataset), that training on diverse sources improves transferability by 1.88–4.05 percentage points with minimal loss of in-domain accuracy, and that Dad Jokes is the best source but worst target of transfer. The paper releases data and code.","tokens_in":16249,"tokens_out":4830,"duration_ms":57313,"significance":"If the reported results survive scrutiny, the paper is a useful first systematic step toward understanding humor transfer in LLMs. Its strengths include a reasonably broad experimental grid over four datasets and two model families, explicit zero-shot baselines, four training seeds with reported standard deviations, and public code/data. The central scientific contribution—that humor competence is partially transferable across types and that data diversity helps—is plausible and potentially actionable for dataset design. However, the headline numerical claims are currently compromised by a hyperparameter-selection protocol that uses the held-out target data, and one of the four datasets relies on GPT-4-generated negatives, which confounds the main type-level asymmetry. Both issues are fixable but require re-analysis.","major_comments":[{"comment":"The reported transfer and diversity numbers are produced by a model-selection protocol that peeks at the held-out target datasets. Appendix D states that the top 3 configurations per dataset are chosen 'based on median cross-validation accuracy across all evaluation datasets', and that 'the configuration with the highest median accuracy across all datasets was selected for final evaluation.' Because 'all datasets' includes the transfer targets, the single/double/triple accuracies are maxima over configurations selected on target validation folds. The candidate pool grows from 3 (single) to 6 (double) to 9 (triple), so the 1.88–4.05 percentage-point diversity gains in Table 7 are confounded with increased selection pressure. The existence of above-chance transfer may well survive a target-blind re-analysis, but the specific magnitudes and the central diversity claim are not supported as r","section":"Appendix D and Section 5.3 / Table 7"},{"comment":"The Dad Jokes 'non-funny' class is generated by GPT-4-Turbo via minimal rewriting of positive jokes. Because GPT-4 is itself an LLM, the rewrites may carry systematic stylistic fingerprints, making the negative class identifiable by artifact rather than by humor structure. This directly affects the paper's most distinctive finding—Dad Jokes as the best transfer source but hardest transfer target—and is consistent with the high 93–94% in-domain accuracies. The manual review of 3,000 outputs checked for style/content preservation and punchline removal, but it did not validate with human annotators that the rewritten texts are actually non-funny, nor did it control for LLM stylistic cues. I request a human unfunniness validation study and, if possible, an additional negative class from naturally non-humorous human-written text, with a comparison of the Dad Jokes transfer results under both","section":"Section 3.1 and Appendix B"},{"comment":"The qualitative hierarchy (e.g., 'Dad Jokes enables strong transfer but is difficult to generalize to' and 'Headlines and One Liners are easier targets') is based on point estimates without statistical testing or confidence intervals. Given the small number of datasets and the selection issue in Appendix D, these pairwise asymmetries should be treated as exploratory unless accompanied by a target-blind protocol and some measure of uncertainty over the selection procedure. This is not a demand for new data, but the claims in Section 5.2 should be softened or explicitly labeled as hypothesis-generating until the re-analysis is available.","section":"Sections 5.1–5.3"}],"minor_comments":[{"comment":"The phrase 'unseen datasets' is currently inaccurate under the reported protocol, since the held-out targets participate in hyperparameter selection. After the re-analysis, define explicitly whether 'unseen' means 'never used at any point in training or validation.'","section":"Abstract / Section 4"},{"comment":"The asterisk on the Amazon length row is explained in the text but would be clearer as a table footnote; the two length statistics for the same row could also be separated into two rows for readability.","section":"Table 1"},{"comment":"Minor typo: 'Nickleback' should be 'Nickelback'. Also, the few-shot prompt contains only five examples; the selection criteria for these examples are not stated.","section":"Appendix B"},{"comment":"The embedding-similarity analysis is based on a single model (Mistral) with no variance estimate. It is framed as exploratory, but the conclusions in the text would benefit from an explicit exploratory caveat.","section":"Section 5.4 / Appendix H"},{"comment":"Clarify which split is used for 'cross-validation accuracy across all evaluation datasets'—validation folds of all four datasets, or only the training dataset? Also state explicitly whether the final test folds were ever used to select hyperparameters.","section":"Appendix D"},{"comment":"The in-domain comparison between single-dataset and triple-dataset training is informative, but the two setups use different selected hyperparameters and different validation protocols; a sentence acknowledging this confound would improve precision.","section":"Section 5.3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is well positioned for the journal if the authors can rerun the analysis with a target-blind hyperparameter-selection rule. I would not require additional datasets; the current four-dataset design is sufficient for a corrigible empirical contribution. My main worry for the editor is that the Appendix D selection protocol, unless changed, will undermine reader trust in the reported magnitudes, even if the qualitative finding of partial transfer is robust. The Dad Jokes negative-class issue is also important but may be addressable with a focused validation study rather than a full re-collection of data."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is the first systematic held-out transfer study across four humor datasets, and the broad pattern—some transfer exists, Dad Jokes is a strong source and weak target—is likely real. But the specific numbers in the abstract (up to 75% accuracy, 1.88–4.05% diversity gains) are not trustworthy at face value because hyperparameter selection peeks at the target datasets.\n\nWhat's new and good: the design—training on one, two, or three datasets and testing on a held-out fourth, with two model families—is a real step beyond Baranov et al. and Arora et al. The dataset embedding similarity analysis and the asymmetric hierarchy are useful. The paper is transparent about limitations and releases code and data. The transfer-existence claim is well supported: most transfer accuracies sit well above the 40–56% zero-shot baselines, even after discounting selection effects.\n\nSoft spots, in rough order of severity:\n\n1. Appendix D: configuration selection uses the held-out target validation sets. \"Median cross-validation accuracy across all evaluation datasets\" means the chosen hyperparameters are selected on the very targets the paper claims are unseen. That biases every reported accuracy, and it biases the diversity comparison because double/triple setups draw from pools of 6 or 9 configs versus 3 for single. The 1.88–4.05% gains could partly be selection pressure, not diversity. This is fixable: select configs on training-domain validation only, or use nested CV.\n\n2. Dad Jokes negatives are generated by GPT-4 Turbo. The paper's signature asymmetry—Dad Jokes hard to transfer to—is partly a statement about detecting GPT-4 rewrite style. The manual check of 3,000 examples covers style/content preservation, not whether the \"unfunny\" judgment matches human judgment. This matters most for interpreting type-level conclusions, less for the overall transfer existence.\n\n3. No significance tests; many differences are within one standard deviation across seeds (Table 6). The paper reports averages without error bars on the headline difference numbers.\n\nOn balance: the central claim \"transfer exists and is asymmetric\" holds up in direction. The magnitude claims do not. This is a useful empirical study worth a serious referee, but it needs re-analysis with training-only selection, human-validated or replaced negatives, and significance testing before the abstract can be taken literally. I'd send it out, and I'd tell the authors to redo model selection before resubmission.","headline":"First systematic held-out humor-transfer study, but target-peeking hyperparameter selection and GPT-4-authored negatives mean the headline numbers are suggestive, not definitive.","tokens_in":16784,"tokens_out":2038,"would_cite":false,"duration_ms":22622,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Humor competence transfers across joke types in LLMs, with Dad Jokes the best source and hardest target.","keywords":["humor detection","transfer learning","multi-task learning","large language models","dataset diversity","Dad Jokes","sarcasm","generalization"],"falsifier":"Replace the GPT-4-generated Dad Jokes negatives with human-written non-funny texts matched for style and content, then rerun the single-dataset and triple-dataset transfer experiments. If the asymmetry (Dad Jokes best source, worst target) and the diversity harm on Dad Jokes disappear or shrink, the original results are artifacts of the generated negative class.","tokens_in":15763,"feed_emoji":"😂","tokens_out":7157,"duration_ms":65103,"temperature":0.7,"pith_summary":"This paper asks whether an LLM trained on one or more types of humor can recognize a humor type it has never seen, and whether such transfer is helped by training on diverse humor sources. The authors fine-tune two 7-billion-parameter LLMs on four humor datasets—Amazon product questions, one-liners, sarcastic news headlines, and Reddit Dad Jokes—in single, paired, and triple combinations, then test each on the held-out humor type. They report that transfer is real and asymmetric: models reach up to 75% accuracy on unseen joke types, training on multiple sources improves transfer by 1.88–4.05 percentage points with only a small in-domain cost, and Dad Jokes are the best enabling source but the hardest target. A reader should care because new humor formats appear constantly online, and the question is whether machines can understand them without retraining on each new format. The paper also contributes a method for converting positive-only joke collections into supervised training sets by having an LLM minimally rewrite jokes into non-funny counterparts.","feed_headline":"Dad jokes teach AI other humor, but resist learning themselves","feed_subtitle":"Mixed-style training lifts accuracy on unseen joke types to 75%; diversity helps most types, hurts Dad Jokes.","key_machinery":"The experimental machinery is a controlled transfer-learning ladder: three training regimes (single, pair, triple dataset) of fixed 5,000-sample size with equal sampling per dataset, tested on each held-out humor type. The asymmetry pattern is measured by cross-accuracy matrices among four binary humor-detection datasets. The Dad Jokes negative class is generated by GPT-4 Turbo making minimal word-level edits to real jokes to remove humor, which is what makes an originally positive-only dataset usable for binary classification and sustains the finding that Dad Jokes is unique as a source and target.","core_discovery":"On the paper's own terms, the central discovery is that humor competence is partially transferable across joke types in instruction-fine-tuned LLMs, and the transfer is structured rather than uniform. Fine-tuning on Amazon questions transfers well to headlines (75%) and one-liners (72%), while the reverse transfers are weaker; Dad Jokes support strong transfer to all other types (68–71%) but are the hardest target for models trained elsewhere (51–62%). Increasing training diversity from one to two datasets improves transfer by 2.02–3.02 points on average (with a further modest gain to three datasets for LLaMA-2), while in-domain accuracy drops only 0.49–1.76 points even though in-domain data","pith_inferences":["If the GPT-4-rewritten negatives are valid non-humor, the same minimal-edit recipe could expand humor-transfer studies to other positive-only sources like caption contests or r/cleanjokes, letting researchers test whether the Dad Jokes asymmetry is general.","The 'best source, worst target' pattern resembles domain generalization in other text tasks: diverse, structurally rich sources transfer broadly while narrow, templatic targets overfit surface conventions—so the same ladder could be run on other 'deep' vs 'shallow' text domains, not just humor.","The result that diversity hurts Dad Jokes transfer, if it replicates, is a practical warning for data-mixing strategies: complex styles may need dedicated in-domain data even when other humor data is abundant.","Because only two 7B models were used and they differed in transfer strength, the claim that the hierarchy reflects humor structure rather than architecture is open: repeating the ladder on larger or newer instruction-tuned models would settle it."],"forward_implications":["LLMs can be prepared for novel humor formats by training on a handful of diverse existing types, without waiting for examples of the new format.","Increasing training-data diversity (even while shrinking per-type data to one third) improves transfer to unseen humor types while keeping in-domain accuracy nearly intact.","Dad Jokes should be included as source data in humor-transfer curricula, but a model that must detect Dad Jokes specifically will need some in-domain examples.","The consistent asymmetry across two model families points to shared structural properties among humor types that could inform cognitive theories of humor.","Positive-only joke datasets can be made usable for binary humor detection via minimal semantic edits that remove the humorous element."],"supporting_citations":[{"why":"Supplies the Amazon product-questions humor dataset, one of the four transfer sources and targets.","marker":"(Ziser et al., 2020)"},{"why":"Supplies the One Liners dataset, the most receptive transfer target.","marker":"(Mihalcea and Strapparava, 2005)"},{"why":"Supplies the Sarcasm Headlines dataset (The Onion vs. real headlines).","marker":"(Misra and Arora, 2023)"},{"why":"Supplies the r/dadjokes positive examples used for the Dad Jokes dataset.","marker":"(Reddit, 2023)"},{"why":"Provides GPT-4 Turbo, used to generate minimal-edit non-funny rewrites for the Dad Jokes negative class.","marker":"(OpenAI, 2023)"},{"why":"Provides LLaMA-2-7B, one of the two fine-tuned models.","marker":"(Touvron et al., 2023)"},{"why":"Provides Mistral-7B, the other fine-tuned model and the better transfer performer.","marker":"(Jiang et al., 2023)"},{"why":"Provides LoRA, the parameter-efficient fine-tuning method used in all experiments.","marker":"(Hu et al., 2022)"}],"fun_headline_variants":["Dad jokes are the best teachers for AI humor, but the worst students","To teach AI new humor, start with Dad jokes—just don't expect it to learn them","AI humor transfer: diverse training helps, Dad jokes lead the way","Dad jokes unlock AI humor transfer, but resist learning themselves","Mixed training helps AI get new joke types; Dad jokes are the key"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The Dad Jokes 'non-funny' examples are made by GPT-4 minimally rewriting real jokes until they are no longer funny, and the paper's most distinctive findings assume those rewrites are valid non-humor rather than recognizable machine-generated text.","fun_headline_variants_meta":{"raw":{"variants":["Dad jokes are the best teachers for AI humor, but the worst students","To teach AI new humor, start with Dad jokes—just don't expect it to learn them","AI humor transfer: diverse training helps, Dad jokes lead the way","Dad jokes unlock AI humor transfer, but resist learning themselves","Mixed training helps AI get new joke types; Dad jokes are the key"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000927,"raw_usage":{"total_tokens":3826,"prompt_tokens":783,"completion_tokens":3043,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":2946}},"tokens_in":527,"tokens_out":3043,"duration_ms":20783,"temperature":1.0,"reasoning_tokens":2946,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:48:39.663369+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the GPT-4-generated Dad Jokes negatives with human-written non-funny texts matched for style and content, then rerun the single-dataset and triple-dataset transfer experiments. If the asymmetry (Dad Jokes best source, worst target) and the diversity harm on Dad Jokes disappear or shrink, the original results are artifacts of the generated negative class.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Amazon product-questions humor dataset, one of the four transfer sources and targets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the One Liners dataset, the most receptive transfer target."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Sarcasm Headlines dataset (The Onion vs. real headlines)."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the r/dadjokes positive examples used for the Dad Jokes dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides GPT-4 Turbo, used to generate minimal-edit non-funny rewrites for the Dad Jokes negative class."}],"review_version":1}