Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Humor Detection: A Transformer Gets the Last Laugh

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning a Transformer on Reddit upvote labels yields a humor detector that outperforms CNN baselines and general human raters on held-out jokes, and transfers to pun and short-joke classification.

desk verdict The Reddit r/Jokes dataset is a real resource, but the 200-upvote threshold and genre-confounded transfer experiments mean the paper shows popularity prediction and joke-style detection more than humor understanding. read the letter →

arxiv 1909.00252 v1 pith:5YHJ3TKJ submitted 2019-08-31 cs.CL cs.LG

Orion Weller , Kevin Seppi This is my paper · ORCID
classification cs.CLcs.LG
keywords humordetectionBERTTransformerRedditr/Jokesjokeclassificationupvotelabelstransferlearningself-attention
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a machine can learn what makes a given online community laugh without any direct human annotation: jokes from Reddit's r/Jokes forum are labeled funny or not by whether their upvote score crosses 200, and a fine-tuned Transformer (BERT) trained on about 16,000 such labeled jokes predicts held-out jokes with 72.4% accuracy. That beats a CNN baseline, which lands in the high 60s, and general human raters, who agreed with the Reddit labels only 66.3% of the time. The same model, without further fine-tuning, identifies puns with an F1 of 93.1% and short jokes with 98.6%, surpassing previously reported CNN-based results. If true, this means humor detection can be reframed as a transferable text-classification task driven by community engagement signals rather than expert annotation.

What carries the argument

The load-bearing mechanism is the Transformer's multi-head self-attention, instantiated in the pre-trained BERT encoder and fine-tuned with a binary classification layer. Its role is to weight words and sentence context so that the model distinguishes high-upvote from low-upvote r/Jokes posts; the training signal is supplied by Reddit's upvote distribution, with a score of 200 as the humor threshold, and by the body/punchline split of each post.

What would settle it

Take held-out jokes whose upvote scores sit just below and just above 200, have a sample of r/Jokes readers rate them for humor, and compare those ratings with the 200-threshold labels; if human humor ratings do not track the threshold, or if the model's accuracy on those near-threshold jokes is at chance, the central claim fails.

Watch

Extended reading notes

Core claim

The central discovery is that the self-attention of a pre-trained Transformer, fine-tuned on community upvote labels, captures enough about joke structure to predict what r/Jokes users will find funny. In experiments with the body alone, the punchline alone, and the full joke, the model reaches 72.4% accuracy on the full held-out set, compared with roughly 68% for the CNN baseline and 66.3% for general human raters. The paper also reports that the punchline contributes more to the prediction than the setup, and that the model transfers to two existing joke-identification benchmarks: Pun of the Day (F1 93.1%) and the Short Jokes dataset (F1 98.6%).

Load-bearing premise

The binary label 'funny' is defined by a Reddit upvote score crossing 200; if upvotes reflect visibility, timing, or community in-crowd effects rather than humor, then the model is predicting an arbitrary score cutoff, not humor.

Editorial extensions

If this is right

  • A binary humor classifier can be built for any community that leaves numeric reactions, with no hand-labeled jokes required.
  • The same model trained on Reddit jokes transfers to pun and short-joke detection without retraining, so joke-likeness learned from one genre helps in another.
  • Because the punchline carries more predictive weight than the setup, humor-detection systems may improve by architecturally emphasizing the final segment of a joke.
  • Reported accuracy depends on a community-specific label, so model scores should be read as predictions of r/Jokes taste, not of universal humor.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Upvotes are a continuous signal, so treating them as a binary above-or-below 200 discards information; a regression or ordinal model on the raw score may distinguish viral from merely funny jokes more sharply.
  • The human baseline came from general raters, not from r/Jokes readers; an accuracy comparison against the target community itself would likely be lower, since the model is tuned to the community's taste.
  • The large gap between the Transformer's accuracy on Reddit jokes (72.4%) and on short jokes (98.6%) suggests the Reddit task is the harder one, likely testing degree of humor within jokes rather than merely whether text is a joke at all.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents a binary humor-detection model built by fine-tuning BERT on Reddit r/Jokes data. Jokes are labeled 'funny' if their upvote score exceeds 200, a threshold the authors describe as a natural divide in the score distribution. The model is trained on the joke body, the punchline, or both; the best configuration reaches 72.4% accuracy on a held-out 25% test split, outperforming a CNN baseline and a Mechanical Turk human-rater baseline. The same model, without further fine-tuning, is then applied to the Pun of the Day and Short Jokes datasets, achieving F1 scores of 93.1% and 98.6%, respectively, which the paper claims surpasses all prior work. The paper's stated contribution is a new task of assessing whether a joke is humorous and a new labeled dataset with open-source code.

Significance. If the results hold, the paper would be a useful application of Transformer models to humor, with the notable practical asset of a publicly released dataset and code. The transfer experiments also compare against the same CNN baseline used in prior work. However, the significance is conditional on the validity of the upvote-threshold label, which the current manuscript does not establish. As presented, the accuracy numbers may measure the ability to predict Reddit popularity rather than humor. The paper's honest acknowledgment of the genre confound in transfer experiments (Section 6) is a strength that should be incorporated into the central claim rather than treated as an aside.

major comments (4)
  1. [Section 3.1] The binary label in the Reddit experiment is defined by a 200-upvote cutoff that was chosen after inspecting the score distribution ('We used this natural divide as the cutoff'). Because scores were collected hourly during March-April 2019 and later updated, the final score conflates humor with exposure time, posting time, and subreddit visibility, and the cutoff itself is never validated against an independent humor judgment. As a result, the 72.4% test accuracy in Table 2 may be measuring the model's ability to predict whether a post crossed an arbitrary popularity threshold rather than whether it is humorous. A concrete fix is to evaluate threshold sensitivity (e.g., report accuracy at several cutoffs) and to validate a subsample of the labels with independent human raters.
  2. [Section 4.2 and Section 5.2] The experiments are based on a single 75/25 stratified split with no confidence intervals, no multiple random seeds, and no significance tests. The headline comparison in Table 2 ('the Transformer achieved an accuracy of 72.4 percent ... while the CNN was in the high 60's') could therefore be within run-to-run variance. For a claim of this strength, repeated resampling or at least a significance test is necessary.
  3. [Sections 3.2, 3.3 and Tables 3-4] The transfer experiments compare jokes against non-joke sentences drawn from news text. The near-ceiling F1 scores (93.1% and 98.6%) may be driven by an easy joke-versus-news genre distinction, a limitation the paper itself acknowledges in Section 6 ('it seems to be an easier task to tell when something isn't a joke at all'). To support the claim that the model understands humor, the negative class should include non-humorous but joke-like text, or the analysis should report performance on a genre-balanced control.
  4. [Table 2 and Section 5.1] The human baseline in Table 2 is reported as '66.3%' with no formal definition. From Section 5.1 it appears this is the proportion of jokes for which the majority of Mechanical Turk raters labeled the joke humorous, not an accuracy against the Reddit labels. If so, it is not directly comparable to the model accuracy and should be either recalculated as agreement with the Reddit-derived labels or clearly labeled as a human agreement rate.
minor comments (5)
  1. [Section 4.1] Figure 1 is a generic reproduction of the Vaswani et al. transformer and does not show the fine-tuned BERT classification head; consider replacing it with a diagram of the actual model or removing it.
  2. [Section 1] The phrase 'the inception ... of the Attention mechanism' is awkward, and the citation to Bahdanau et al. (2015) does not obviously support the surrounding historical claim; consider rewording and correcting the reference context.
  3. [Section 3.1] The paper does not report the posting dates of the collected jokes or a minimum age for posts entering the dataset; this information is needed to evaluate the exposure-time confound discussed in the major comments.
  4. [Section 3.2] The paper states that the Short Jokes negative set was built using 'the same method and news crawl source' as Chen and Soo (2018), but the exact sentence-matching criteria are not specified; please provide the matching details or a pointer to code.
  5. [References] There are several typographical errors in the reference list (e.g., 'V on-Wun Soo' and 'Proceedings of the IEEE ,'), which should be corrected in a revision.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: held-out supervised evaluation and independent transfer benchmarks; the upvote threshold is a construct-validity concern, not a circular step.

full rationale

The paper contains no derivation chain in which a prediction is algebraically or statistically forced by its inputs. The Reddit experiment is conventional supervised classification: jokes are labeled by an upvote-threshold rule ('We used this natural divide as the cutoff to decide what qualified as a funny joke'), the data are split 75/25 with stratified labels, and the reported 72.4% accuracy is on a held-out test set. The threshold is a label-construction choice, not a model parameter; nothing in BERT's training or inference makes the test prediction equal to the threshold. The transfer experiments are external: Pun of the Day was scraped by Yang et al., and Short Jokes is a Kaggle dataset with WMT16 negative sentences; the model is applied without further fine-tuning and compared against previously reported CNN baselines. These benchmarks are independent of the Reddit labels. The MTurk human ratings provide an independent, albeit imperfect, check on the Reddit labels. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via citation. The only close concern is whether 'funny equals upvote score above 200' is a valid operationalization of humor; that is a measurement and construct-validity issue, not circularity, because the model's output is not equivalent to the threshold by construction. Accordingly, no circular step is identified.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on an operational definition of humor (upvotes over a threshold), a single post hoc cutoff, and negative examples from news text for transfer evaluation. These are domain assumptions rather than fitted equations, but they carry much of the weight of the reported results.

free parameters (4)
  • Upvote threshold for humor label = 200 upvotes
    Chosen post hoc from the observed score distribution ('major jump between the 0-200 upvote range and the 200 range and onwards') to create the binary labels; it determines both training and test labels.
  • BERT fine-tuning learning rate = 2e-05
    Manual choice, not fitted to data; standard for BERT fine-tuning.
  • Max sequence length = 128 tokens
    Manual choice; jokes longer than 128 tokens are truncated.
  • Training epochs = up to 7 with checkpoints
    Early-stopping style choice; exact checkpoint selection is not specified.
assumptions (4)
  • domain assumption Reddit upvote counts are a faithful measure of humor for r/Jokes users.
    The entire dataset and label construction rely on this (§3.1); if upvotes reflect popularity, timing, or other factors, the labels are noisy.
  • ad hoc to paper The 200-upvote threshold separates funny from not funny jokes.
    Introduced after inspecting the distribution, not justified by an external criterion (§3.1).
  • domain assumption News-crawl sentences are valid negative examples for humor detection in Puns and Short Jokes datasets.
    The transfer benchmarks compare jokes or puns against news text (§3.2, §3.3), so the model may learn genre differences rather than humor.
  • domain assumption Human Mechanical Turk ratings are a meaningful comparison baseline.
    Human raters were asked whether they personally find each joke humorous, producing 66.3% positive ratings, which is compared to the model's binary accuracy (§5.1, Table 2). This conflates a rating task with a classification task.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Humor Detection: A Transformer Gets the Last Laugh." pith.science (2026). https://pith.science/paper/5YHJ3TKJ

@misc{pith2026190900252,
  author       = {Pith},
  title        = {Pith review of: Humor Detection: A Transformer Gets the Last Laugh},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5YHJ3TKJ}},
  note         = {Machine review of arXiv:1909.00252}
}
read the original abstract

Much previous work has been done in attempting to identify humor in text. In this paper we extend that capability by proposing a new task: assessing whether or not a joke is humorous. We present a novel way of approaching this problem by building a model that learns to identify humorous jokes based on ratings gleaned from Reddit pages, consisting of almost 16,000 labeled instances. Using these ratings to determine the level of humor, we then employ a Transformer architecture for its advantages in learning from sentence context. We demonstrate the effectiveness of this approach and show results that are comparable to human performance. We further demonstrate our model's increased capabilities on humor identification problems, such as the previously created datasets for short jokes and puns. These experiments show that this method outperforms all previous work done on these tasks, with an F-measure of 93.1% for the Puns dataset and 98.6% on the Short Jokes dataset.

Figures

Figures reproduced from arXiv: 1909.00252 by the authors.

Figure 1
Figure 1. Transformer Model Architecture self attention layer, followed by a feed-forward network. It then outputs the information into the decoder, which includes the previously mentioned layers, plus an additional masked attention step. Afterwords, it is transformed through a softmax into the output. This model’s success is in large part due to the Transformer’s self-attention layers. We chose a learning rate of 2e-05 and a… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CleanComedy: Creating Friendly Humor through Generative Techniques

    cs.CL 2024-12 conditional novelty 6.0 of 10

    CleanComedy delivers cleaner bilingual joke datasets, but its fine-tuned models produce less funny jokes than GPT-4o or unfiltered human jokes.

  2. Decoders Laugh as Loud as Encoders

    cs.CL 2025-09 conditional novelty 5.0 of 10

    Fine-tuned GPT-4o classified jokes into five humor categories plus no-joke with macro-F1 0.852, statistically indistinguishable from RoBERTa-base's 0.857 on a small custom dataset.

Reference graph

Works this paper leans on

21 extracted references · 17 canonical work pages · cited by 2 Pith papers

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1 'mid.sentence := #2 '...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Dzmitry Bahdanau , Kyunghyun Cho , and Yoshua Bengio . 2015. Neural Machine Translation by Jointly Learning to Align and Translate . International Conference on Learning Representations

    work page 2015

  4. [4]

    Hareesh Bahuleyan . 2018. Natural Language Generation with Neural Variational Models . arXiv e-prints, page arXiv:1808.09012

    work page Pith review arXiv 2018

  5. [5]

    Peng-Yu Chen and Von-Wun Soo. 2018. Humor recognition using deep learning. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)

    work page 2018

  6. [6]

    Jacob Devlin , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding . North American Chapter of the Association for Computational Linguistics

    work page 2018

  7. [7]

    He He, Nanyun Peng, and Percy Liang. 2019. Pun generation with surprise. In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 1734--1744, Minneapolis, Minnesota. Association for Computational Linguistics

    work page 2019

  8. [8]

    Yann LeCun, Leon Bottou, Y Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86:2278 -- 2324

    work page 1998

Show all 21 references
  1. [9]

    Liu , Mohammad Saleh , Etienne Pot , Ben Goodrich , Ryan Sepassi , Lukasz Kaiser , and Noam Shazeer

    Peter J. Liu , Mohammad Saleh , Etienne Pot , Ben Goodrich , Ryan Sepassi , Lukasz Kaiser , and Noam Shazeer . 2018. Generating Wikipedia by Summarizing Long Sequences . International Conference on Learning Representations

    2018

  2. [10]

    Minh-Thang Luong , Hieu Pham , and Christopher D. Manning . 2015. Effective Approaches to Attention-based Neural Machine Translation . Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages pages 1412 -- 1421

    2015

  3. [11]

    Rada Mihalcea and Carlo Strapparava. 2005. Making computers laugh: Investigations in automatic humor recognition. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, HLT '05, pages 531--538, Stroudsburg, PA, USA. ...

    2005

  4. [12]

    Amruta Purandare and Diane Litman. 2006. Humor: Prosody analysis and automatic recognition for f*r*i*e*n*d*s*. Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 208--215

    2006

  5. [13]

    Yishay Raz. 2012. Automatic humor classification on twitter. Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop, pages 66--70

    2012

  6. [14]

    Rupesh Kumar Srivastava , Klaus Greff , and J \"u rgen Schmidhuber . 2015. Highway Networks . arXiv e-prints, page arXiv:1505.00387

    2015 arXiv

  7. [15]

    Taylor and Lawrence J

    Julia M. Taylor and Lawrence J. Mazlack. 2004. Computationally recognizing wordplay in jokes. In Proceedings of CogSci 2004

    2004

  8. [16]

    let everything turn well in your wife

    Alessandro Valitutti, Hannu Toivonen, Antoine Doucet, and Jukka M Toivanen. 2013. “let everything turn well in your wife”: Generation of adult humor using lexical constraints. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2:...

    2013

  9. [17]

    Gomez , Lukasz Kaiser , and Illia Polosukhin

    Ashish Vaswani , Noam Shazeer , Niki Parmar , Jakob Uszkoreit , Llion Jones , Aidan N. Gomez , Lukasz Kaiser , and Illia Polosukhin . 2017. Attention Is All You Need . 31st Conference on Neural Information Processing Systems, pages 17--21

    2017

  10. [18]

    Kelvin Xu , Jimmy Ba , Ryan Kiros , Kyunghyun Cho , Aaron Courville , Ruslan Salakhutdinov , Richard Zemel , and Yoshua Bengio . 2015. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention . arXiv e-prints, page arXiv:1502.03044

    2015 arXiv

  11. [19]

    Diyi Yang, Alon Lavie, Chris Dyer, and Eduard Hovy. 2015. Humor recognition and humor anchor extraction. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2367--2376

    2015

  12. [20]

    Smola, and Eduard H

    Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alexander J. Smola, and Eduard H. Hovy. 2016. Hierarchical attention networks for document classification. HLT-NAACL

    2016

  13. [21]

    Renxian Zhang and Naishi Liu. 2014. Recognizing humor on twitter. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, CIKM '14, pages 889--898, New York, NY, USA. ACM

    2014

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.