Decoders Laugh as Loud as Encoders

Eli Borodach; Rajat Dandekar; Raj Dandekar; Sreedath Panat

REVIEW 3 major objections 6 minor 10 references

Decoders Laugh as Loud as Encoders

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A fine-tuned GPT-4o reaches a mean macro-F1 of 0.8522 on six-way English humor classification, statistically tied with the best fine-tuned encoder, RoBERTa-base at 0.8566.

desk verdict Useful small benchmark showing fine-tuned GPT-4o can roughly match RoBERTa on one humor dataset, but the 'parity' claim is statistically under-supported. read the letter →

arxiv 2509.04779 v1 pith:QNL4WQI7 submitted 2025-09-05 cs.CL cs.AI

Eli Borodach , Raj Dandekar , Rajat Dandekar , Sreedath Panat This is my paper · ORCID

classification cs.CLcs.AI

keywords humorclassificationlargelanguagemodelsdecoder-onlyencoder-onlyfine-tuningGPT-4oRoBERTamacro-F1

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether large language models understand humor, and answers by testing six-way humor classification as a proxy: five joke types (absurdity, dark, irony, wordplay, social commentary) plus a no-joke class. It reports that a fine-tuned decoder-only model, GPT-4o, reaches a mean macro-F1 of $0.8522 \pm 0.0056$ on its 1,392-example dataset, while the best fine-tuned encoder, RoBERTa-base, reaches $0.8566 \pm 0.0164$; a Welch's t-test finds no significant difference. The central claim is that a decoder built for text generation, not classification, can match the best encoder on this comprehension-style task once fine-tuned. That matters because it weakens the common assumption that encoder-only models are the natural choice for understanding-oriented NLP tasks, and it opens humor classification to the simpler decoder fine-tuning pipeline. The paper frames classification performance as evidence of humor understanding, a proxy it states explicitly from the start.

What carries the argument

The load-bearing object is the evaluation setup and its statistical comparison rather than a new theorem. The dataset contains 1,392 examples across six classes, with a stratified 80/10/10 split; macro-F1, the unweighted average of per-class F1 scores, is the metric, chosen so that the abundant no-joke class (697 examples) cannot mask the rare social-commentary class (62 examples). Encoders were fine-tuned for 20 epochs with best-epoch selection over three seeds, while GPT-4o was fine-tuned for 3 epochs with a batch of 4 and greedy decoding, also over three seeds. Welch's t-test on the three-run macro-F1 distributions is what turns a small mean gap of $0.0044$ into a claim of statistical parity.

What would settle it

Have several independent annotators re-label the 1,392 examples and compare each model against the majority-vote labels: if fine-tuned GPT-4o no longer matches RoBERTa on that relabeled set, the reported parity is an artifact of a single noisy labeling; if the original labels themselves agree poorly across annotators, the proxy for humor understanding fails regardless of model scores.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a parity result: a fine-tuned decoder-only model classifies English humor into six categories as accurately as the best fine-tuned encoder. GPT-4o attains a mean macro-F1 of $0.8522 \pm 0.0056$ across three seeded runs, and RoBERTa-base attains $0.8566 \pm 0.0164$; Welch's t-test gives $p > 0.05$, so the difference is not statistically significant. The paper's proxy, stated in Section 1, is that correct classification implies the model understands the jokes. It contrasts this with the earlier literature, where the most recent decoder checked for humor was GPT-2 and where decoder-only models were thought to lag behind encoders. The paper also reports that zero-shot and few-shot decoders all scored below $0.60$ macro-F1, which isolates fine-tuning as the step that produces the parity.

Load-bearing premise

The conclusion stands or falls on treating a model's ability to reproduce the dataset's human-assigned humor-type labels as evidence that the model understands the jokes; if those labels are unreliable or the non-joke sentences are not representative of ordinary English, the equality of the two models' scores says little about humor understanding.

Editorial extensions

If this is right

Fine-tuned decoder-only models can serve as humor classifiers without an encoder head, because GPT-4o's macro-F1 is statistically indistinguishable from RoBERTa-base's.
The usual assumption that encoders are better at understanding-oriented tasks is weakened for humor: a model trained to generate text matches the best encoder once fine-tuned.
Fine-tuning is the mechanism that unlocks decoder parity: every decoder tested in zero-shot or few-shot mode stayed below 0.60 macro-F1, while fine-tuned GPT-4o reached 0.85.
Larger encoders do not uniformly help on this task, so the practical baseline for six-way humor classification remains RoBERTa-base, not a larger variant.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

We infer that the parity result is sensitive to label quality; a direct test would be to re-annotate the same 1,392 examples with several human judges and check whether both models match the majority labels as closely as they match the original single labels.
We infer that 'understanding humor' likely extends beyond classification: the paper's proxy does not test whether a model can explain why a joke is funny, generate a new joke of a specified type, or rank jokes by funniness, so parity on categorization may not carry over to those abilities.
We infer that a learning-curve experiment would clarify the result: fine-tuning the decoder on 10%, 25%, and 50% of the training data and comparing with RoBERTa would show whether the decoder's parity is an architectural property or simply a consequence of having enough supervised examples.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a small, honest empirical benchmark showing that a fine-tuned GPT-4o roughly matches RoBERTa on a six-way humor classification task. The result is plausible and mildly interesting as a model-selection data point, but the paper overstates the 'parity' claim, and the dataset is too small and under-reported to be an authoritative comparison.

What's new: previous work had looked at GPT-2/GPT-3 humor generation and classification with encoders; nobody had fine-tuned GPT-4o as a six-way humor classifier and compared it against a battery of encoders. The paper does that carefully in one sense: 17 encoders, multiple seeds, mean±std, macro-F1 because the classes are imbalanced. The 0.8522 vs 0.8566 numbers are internally consistent, and the t-test result is plausible. They also did some thoughtful data cleaning, replacing category-clue words to reduce shortcut learning.

The soft spots are real. First, the headline claim that GPT-4o 'laughs as loud' as RoBERTa rests on p>0.05 from three runs per model. That's a non-significant difference, not evidence of equivalence. With n=3, the test has low power; the confidence interval for the difference is wide and easily includes a gap several times the observed 0.0044. The paper never reports an equivalence margin, TOST, or a CI for the difference. That's the load-bearing weakness. Second, the dataset is 1,392 examples, manually labeled with no inter-annotator agreement, and the no-joke class is sampled from a different Kaggle corpus. The authors state the proxy assumption explicitly in Section 1, but they don't validate it. Third, no data or code is released, so the result can't be independently checked.

The paper is not a waste of time. The question—can a fine-tuned decoder match the best fine-tuned encoder on a classification task—is legitimate, and the answer here is a useful data point. But it's a data point, not a definitive parity result. The limitations section is honest, and the authors seem aware of most of these issues.

My recommendation: send to peer review with requests for major revision. The referee should ask for (a) an equivalence test or at least a confidence interval on the difference, (b) release of the dataset or use of a public benchmark, and (c) some evidence on label reliability. If those are addressed, the paper would be a solid short contribution.

Referee Report

3 major / 6 minor

Summary. The paper reports a six-way English humor classification benchmark (absurdity, dark, irony, wordplay, social commentary, and no-joke) built from 1,392 manually curated examples, and compares a large set of encoder-only, encoder-decoder, and decoder-only models. The headline result is that a fine-tuned GPT-4o reaches macro-F1 0.8522±0.0056, which the authors describe as statistically indistinguishable (Welch's t-test, p>0.05) from the best encoder, RoBERTa-base, at 0.8566±0.0164. The paper interprets this as evidence that fine-tuned decoder-only models can match fine-tuned encoders on humor classification.

Significance. If the parity result were properly established, it would be a moderately interesting empirical data point, since decoder-only models are often assumed to be weaker at classification tasks. The paper also provides a broad comparison across many models with means and standard deviations over three seeds, and it honestly acknowledges several data limitations. However, the significance is constrained by the small, manually labeled dataset and by the fact that the statistical analysis as reported does not actually demonstrate equivalence. The paper is a useful replication-style contribution only after the statistical and data-validity issues are addressed.

major comments (3)

[§4 (Results), Table 3] The conclusion that GPT-4o 'performs as well as' RoBERTa is not supported by the reported Welch's t-test. With n=3 runs per model, p>0.05 merely means the data cannot reject equality; it does not establish equivalence. The reported means and standard deviations imply a 95% confidence interval for the difference on the order of ±0.04 macro-F1, which includes differences many times larger than the observed 0.0044 gap. The paper should report an equivalence margin, a two one-sided tests (TOST) procedure, or at minimum a confidence interval for the difference, and should avoid phrasing p>0.05 as evidence of parity.
[§3.1–3.3, §6] The construct validity of the benchmark is load-bearing for the humor-understanding interpretation. Labels were assigned manually by the authors with no inter-annotator agreement statistics, and the no-joke class is drawn from a different Kaggle corpus than the joke sources. The paper itself acknowledges in Limitations that the data are scarce and heterogeneous. Without label reliability evidence or an external validation, the parity result is a statement about this custom dataset, not about humor understanding. Please report agreement statistics if available, release the cleaned data, or explicitly restrict the claim to this dataset.
[§3.3, §4] Because the no-joke class constitutes half of the dataset (697 of 1,392 examples) and is sampled from a Kaggle humor-detection corpus, the paper should describe the sampling criteria and any deduplication or filtering applied to those sentences. If the 'regular sentences' are not representative of ordinary non-humorous English, the six-class macro-F1 can be dominated by an artifact of the negative class rather than by humor understanding.

minor comments (6)

[Abstract] 'Allen Turing' should be 'Alan Turing'.
[§2.3] 'Leap-of-Thoguht' is a typo for 'Leap-of-Thought'.
[Table 3 footnote] The entry '0.0.0109' should read '0.0109'.
[§6] The Limitations section says 1394 examples, while Table 2 totals 1392; please reconcile these numbers.
[Table 3] The caption refers to bold and underlined scores, but the table as typeset does not show them; please ensure the formatting is visible.
[Reproducibility] No link to code or to the cleaned dataset is provided; releasing these would improve reproducibility, especially given the manual cleaning steps described in §3.2.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical benchmark whose predictions are measured against externally labeled data, not derived from its own assumptions.

full rationale

The paper reports a conventional supervised classification benchmark: it collects 1,392 English examples from external joke sources and a Kaggle no-joke corpus, manually cleans and labels them, then fine-tunes encoders and GPT-4o on the same train/validation/test split. The central comparison is empirical: mean macro-F1 for GPT-4o (0.8522±0.0056) versus RoBERTa-base (0.8566±0.0164), with Welch's t-test used to support a claim of parity. Nothing in this chain is self-definitional: the labels are not defined by the models' outputs, the metric is not fitted from the predictions, and no equation reduces a predicted quantity to an input parameter. The only mild concern is the paper's own operationalization of humor understanding as classification accuracy, and the low statistical power of a three-run t-test for establishing equivalence, but those are validity and statistical-inference concerns, not circularity. There is no load-bearing self-citation, no imported uniqueness theorem, and no ansatz smuggled in via prior author work. The paper is self-contained as an empirical comparison against external data sources, so the circularity score is 0.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claim rests on the assumption that macro-F1 on this dataset measures humor understanding, on the reliability of manually assigned labels, and on a set of hand-chosen training hyperparameters. There are no invented entities and no formal derivation.

free parameters (5)

GPT-4o fine-tuning epochs = 3
Chosen by hand without reported search; Figure 3 shows training loss reaches zero quickly, indicating overfitting risk.
GPT-4o learning rate multiplier = 2
Chosen by hand; no search reported, affects optimization and final weights.
Encoder fine-tuning epochs = 20
Chosen by hand; best epoch selected by validation macro-F1, which makes the reported test scores conditional on this selection rule.
Random seeds = 42, 1337, 2025
Three seeds define the runs; the parity conclusion rests on only three runs per model.
GPT-4o inference temperature = 0
Greedy decoding chosen for reproducibility.

assumptions (6)

domain assumption Classification accuracy on this dataset is a valid proxy for humor understanding.
Stated in Section 1; the title and conclusions depend on this proxy.
domain assumption The manually assigned category labels are reliable even though no inter-annotator agreement is reported.
Sections 3.1 through 3.2; labels are assigned by the authors without validation.
domain assumption The five-category humor taxonomy plus no-joke is a meaningful decomposition of English humor.
Section 3.1; categories are taken from joke websites and manually filtered.
ad hoc to paper Removing jokes that mix wordplay with another category produces a valid single-label dataset.
Section 3.2.1; this design choice may make categories more separable than in natural humor.
ad hoc to paper Replacing repeated words and category-clue words preserves the humor category label.
Sections 3.2.2 and 3.2.3; the text is altered, which could change the humor type.
domain assumption The Kaggle non-joke sentences are representative of non-humorous English.
Section 3.3; no distributional matching with the joke texts is reported.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Decoders Laugh as Loud as Encoders." pith.science (2026). https://pith.science/paper/QNL4WQI7

@misc{pith2026250904779,
  author       = {Pith},
  title        = {Pith review of: Decoders Laugh as Loud as Encoders},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QNL4WQI7}},
  note         = {Machine review of arXiv:2509.04779}
}

read the original abstract

From the dawn of the computer, Allen Turing dreamed of a robot that could communicate using language as a human being. The recent advances in the field of Large Language Models (LLMs) shocked the scientific community when a single model can apply for various natural language processing (NLP) tasks, while the output results are sometimes even better than most human communication skills. Models such as GPT, Claude, Grok, etc. have left their mark on the scientific community. However, it is unclear how much these models understand what they produce, especially in a nuanced theme such as humor. The question of whether computers understand humor is still open (among the decoders, the latest to be checked was GPT-2). We addressed this issue in this paper; we have showed that a fine-tuned decoder (GPT-4o) performed (Mean F1-macro score of 0.85) as well as the best fine-tuned encoder (RoBERTa with a Mean of F1-score 0.86)

Figures

Figures reproduced from arXiv: 2509.04779 by the authors.

**Figure 1.** Data Collection and Processing Pipeline Absurdity Jokes—humor based on nonsensical or illogical scenarios that defy common sense—were sourced from (UpJoke n.d.[a]) (58 jokes), (Reader’s Digest n.d.[a]) (13), (Reddit n.d.) (21), and (Today n.d.) (24). Dark Jokes—humor involving taboo, morbid, or tragic subjects presented in a humorous way—were scraped from (TheCoolist n.d.) (115). Irony Jokes—humor that relies on sta… view at source ↗

**Figure 2.** Training loss of models over time (steps) [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Training and validation loss progression for GPT-4o fine-tuning [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

10 extracted references · 8 canonical work pages

[1]

On the Identification of Humor Markers in Computer-Mediated Communication

Adams, Audrey Claire (2012). “On the Identification of Humor Markers in Computer-Mediated Communication.” In:AAAI Fall Symposium: Artificial Intelligence of Humor, pp. 2–6. Agarwal, Nidhi et al. (2020). “Transfer learning: Survey and classification”. In:Smart Innovations in Communication and Computational Sciences: Proceedings of ICSICCS 2020, pp. 145–155...

work page 2012
[3]

Is this a joke? detecting humor in spanish tweets

IOP Publishing, p. 012072. Castro, Santiago et al. (2016). “Is this a joke? detecting humor in spanish tweets”. In:Ibero-american conference on artificial intelligence. Springer, pp. 139–150. Chen, Huan-Yu, Yun-Shao Lin, and Chi-Chun Lee (2021). “Through the words of viewers: using comment-content entangled network for humor impression recognition”. In:20...

work page 2016
[10]

Research on Humor Recognition

IEEE, pp. 272–275. Zhang, Donghai, Wei Song, Xianjun Liu, Lizhen Liu, et al. (2018). “Research on Humor Recognition”. In:2018 IEEE 9th international conference on software engineering and service science (ICSESS). IEEE, pp. 152–155. Zhang, Renxian and Naishi Liu (2014). “Recognizing humor on twitter”. In:Proceedings of the 23rd ACM interna- tional confere...

work page 2018
[35]

UR-FUNNY: A multimodal language dataset for understanding humor

14, pp. 12972–12980. Hasan, Md Kamrul, Wasifur Rahman, et al. (2019). “UR-FUNNY: A multimodal language dataset for understanding humor”. In:arXiv preprint arXiv:1904.06618. Hossain, Nabil, John Krumm, and Michael Gamon (2019). “”President Vows to Cut¡ Taxes¿ Hair”: Dataset and Analysis of Creative Text Editing for Humorous Headlines”. In:arXiv preprint ar...

arXiv 2019
[43]

Deceptive Humor: A Synthetic Multilingual Benchmark Dataset for Bridging Fabricated Claims with Humorous Content

Kamal, Ashraf and Muhammad Abulaish (2019). “Self-deprecating humor detection: A machine learning approach”. In:International conference of the pacific association for computational linguistics. Springer, pp. 483–494. Kao, Justine T, Roger Levy, and Noah D Goodman (2016). “A computational model of linguistic humor in puns”. In:Cognitive science40.5, pp. 1...

work page Pith review arXiv 2019
[53]

The linear organization of jokes: analysis of two thousand texts

Attardo, Salvatore et al. (1994). “The linear organization of jokes: analysis of two thousand texts”. In. Barbieri, Francesco and Horacio Saggion (2014). “Automatic Detection of Irony and Humour in Twitter.” In:ICCC, pp. 155–162. Bertero, Dario and Pascale Fung (2016). “Multimodal deep neural nets for detecting humor in TV sitcoms”. In:2016 IEEE Spoken La...

work page 1994
[97]

Word sense and subjectivity

Wiebe, Janyce and Rada Mihalcea (2006). “Word sense and subjectivity”. In:Proceedings of the 21st International conference on computational linguistics and 44th annual meeting of the Association for Computational Linguistics, pp. 1065–1072. Yang, Diyi, Alon Lavie, et al. (2015). “Humor recognition and humor anchor extraction”. In:Proceedings of the 2015 c...

work page 2006
[117]

Humor knowledge enriched transformer for understanding multimodal humor

Hasan, Md Kamrul, Sangwu Lee, et al. (2021). “Humor knowledge enriched transformer for understanding multimodal humor”. In:Proceedings of the AAAI conference on artificial intelligence. Vol

work page 2021

Show all 10 references

[415]

Humor identification using affect based content in target text

— (2020). “Humor identification using affect based content in target text”. In:Journal of Intelligent & Fuzzy Systems 39.1, pp. 697–708. Mihalcea, Rada and Stephen Pulman (2007). “Characterizing humour: An exploration of features in humorous texts”. In:International Conference...

2020 arXiv
[2018]

Multimodal humor dataset: Predicting laughter tracks for sitcoms

co-located with 34th conference of the Spanish society for natural language processing (SEPLN 2018), pp. 204–213. Patro, Badri N et al. (2021). “Multimodal humor dataset: Predicting laughter tracks for sitcoms”. In:Proceedings of the IEEE/CVF winter conference on applications ...

2021 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

On the Identification of Humor Markers in Computer-Mediated Communication

Adams, Audrey Claire (2012). “On the Identification of Humor Markers in Computer-Mediated Communication.” In:AAAI Fall Symposium: Artificial Intelligence of Humor, pp. 2–6. Agarwal, Nidhi et al. (2020). “Transfer learning: Survey and classification”. In:Smart Innovations in Communication and Computational Sciences: Proceedings of ICSICCS 2020, pp. 145–155...

work page 2012

[2] [3]

Is this a joke? detecting humor in spanish tweets

IOP Publishing, p. 012072. Castro, Santiago et al. (2016). “Is this a joke? detecting humor in spanish tweets”. In:Ibero-american conference on artificial intelligence. Springer, pp. 139–150. Chen, Huan-Yu, Yun-Shao Lin, and Chi-Chun Lee (2021). “Through the words of viewers: using comment-content entangled network for humor impression recognition”. In:20...

work page 2016

[3] [10]

Research on Humor Recognition

IEEE, pp. 272–275. Zhang, Donghai, Wei Song, Xianjun Liu, Lizhen Liu, et al. (2018). “Research on Humor Recognition”. In:2018 IEEE 9th international conference on software engineering and service science (ICSESS). IEEE, pp. 152–155. Zhang, Renxian and Naishi Liu (2014). “Recognizing humor on twitter”. In:Proceedings of the 23rd ACM interna- tional confere...

work page 2018

[4] [35]

UR-FUNNY: A multimodal language dataset for understanding humor

14, pp. 12972–12980. Hasan, Md Kamrul, Wasifur Rahman, et al. (2019). “UR-FUNNY: A multimodal language dataset for understanding humor”. In:arXiv preprint arXiv:1904.06618. Hossain, Nabil, John Krumm, and Michael Gamon (2019). “”President Vows to Cut¡ Taxes¿ Hair”: Dataset and Analysis of Creative Text Editing for Humorous Headlines”. In:arXiv preprint ar...

arXiv 2019

[5] [43]

Deceptive Humor: A Synthetic Multilingual Benchmark Dataset for Bridging Fabricated Claims with Humorous Content

Kamal, Ashraf and Muhammad Abulaish (2019). “Self-deprecating humor detection: A machine learning approach”. In:International conference of the pacific association for computational linguistics. Springer, pp. 483–494. Kao, Justine T, Roger Levy, and Noah D Goodman (2016). “A computational model of linguistic humor in puns”. In:Cognitive science40.5, pp. 1...

work page Pith review arXiv 2019

[6] [53]

The linear organization of jokes: analysis of two thousand texts

Attardo, Salvatore et al. (1994). “The linear organization of jokes: analysis of two thousand texts”. In. Barbieri, Francesco and Horacio Saggion (2014). “Automatic Detection of Irony and Humour in Twitter.” In:ICCC, pp. 155–162. Bertero, Dario and Pascale Fung (2016). “Multimodal deep neural nets for detecting humor in TV sitcoms”. In:2016 IEEE Spoken La...

work page 1994

[7] [97]

Word sense and subjectivity

Wiebe, Janyce and Rada Mihalcea (2006). “Word sense and subjectivity”. In:Proceedings of the 21st International conference on computational linguistics and 44th annual meeting of the Association for Computational Linguistics, pp. 1065–1072. Yang, Diyi, Alon Lavie, et al. (2015). “Humor recognition and humor anchor extraction”. In:Proceedings of the 2015 c...

work page 2006

[8] [117]

Humor knowledge enriched transformer for understanding multimodal humor

Hasan, Md Kamrul, Sangwu Lee, et al. (2021). “Humor knowledge enriched transformer for understanding multimodal humor”. In:Proceedings of the AAAI conference on artificial intelligence. Vol

work page 2021