{"id":"70edbea8-1b15-4aef-8296-e3b7cce65ae6","arxiv_id":"1909.00252","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fine-tuning BERT on Reddit upvote labels yields 72.4% accuracy on held-out r/Jokes jokes and F1 of 93.1% on puns and 98.6% on short jokes, outperforming prior humor-detection models.","lead":"This paper trains a BERT transformer to predict whether a joke is humorous, using Reddit upvotes as labels, and reports accuracy above human raters and previous models. It also shows the same model transfers to two existing joke datasets, with high F1 scores. Why read: it introduces a new 16,000-joke rating dataset and applies modern NLP methods to a subjective social task.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on an unvalidated post hoc upvote threshold: the 200-vote cutoff may measure Reddit popularity rather than humor, so the reported accuracy is not yet evidence of humor detection.","rationale":"The reader's weakest assumption correctly identifies the validity of the binary humor label as the most load-bearing concern. The Reddit upvote threshold is chosen post hoc, is not independently validated, and is confounded with the time a joke had to accumulate votes. If the label does not track humor, the model's 72.4% accuracy is merely predicting a popularity cutoff, and the central claim of humor detection collapses. This concern is addressable: an independent human annotation study on a subset of the test set, or a threshold sensitivity analysis, would settle whether the labels are meaningful. The transfer results on Puns and Short Jokes, while also concerning due to genre confounding (jokes versus news), are secondary to the Reddit task; they support the model's generalization but not the validity of the primary humor labels. Since the issue is fundamental but fixable, the reader's CONDITIONAL verdict is appropriate, and no change is needed. The concrete test would distinguish a genuine humor signal from a popularity artifact.","tokens_in":5691,"tokens_out":6598,"duration_ms":62683,"concrete_test":"Select a random sample of 500 jokes from the Reddit test set, have at least 3 independent annotators rate each as 'humorous' or 'not humorous,' and compare the majority human label to both the upvote-threshold label and the BERT model's prediction. If the model's accuracy against the human majority is substantially below the reported 72.4% (e.g., near chance), while its accuracy against the upvote threshold remains high, then the upvote threshold is not a valid humor signal. As a complementary check, retrain the model with thresholds at 100, 500, and 1000 upvotes; stable performance across thresholds would support the cutoff, while large variation would show the result is an artifact of the specific threshold.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's core experiment labels a joke as humorous iff its Reddit score exceeds 200 upvotes, a cutoff chosen after inspecting the score distribution ('We used this natural divide as the cutoff'). This threshold is post hoc and not validated against any independent humor judgment. Because scores were collected hourly over two months and updated, the final score reflects exposure time and subreddit visibility as much as humorousness; a joke posted late in the collection window has less time to cross the cutoff, so the label is confounded with age and popularity. With only 6% of scores between 200 and 20,000, the 'natural divide' describes the upvote distribution, not humor. Thus the model may be predicting which posts are popular on r/Jokes rather than which are humorous. The transfer results on Puns and Short Jokes are also genre-confounded: negative examples come from news text, so high F1 may reflect joke-versus-news style classification, not humor understanding. The paper itself acknowledges this ('it seems to be an easier task to tell when something isn't a joke at all'), which is a limitation that should be weighed in the verdict.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a binary humor-detection model built by fine-tuning BERT on Reddit r/Jokes data. Jokes are labeled 'funny' if their upvote score exceeds 200, a threshold the authors describe as a natural divide in the score distribution. The model is trained on the joke body, the punchline, or both; the best configuration reaches 72.4% accuracy on a held-out 25% test split, outperforming a CNN baseline and a Mechanical Turk human-rater baseline. The same model, without further fine-tuning, is then applied to the Pun of the Day and Short Jokes datasets, achieving F1 scores of 93.1% and 98.6%, respectively, which the paper claims surpasses all prior work. The paper's stated contribution is a new task of assessing whether a joke is humorous and a new labeled dataset with open-source code.","tokens_in":5926,"tokens_out":3978,"duration_ms":36456,"significance":"If the results hold, the paper would be a useful application of Transformer models to humor, with the notable practical asset of a publicly released dataset and code. The transfer experiments also compare against the same CNN baseline used in prior work. However, the significance is conditional on the validity of the upvote-threshold label, which the current manuscript does not establish. As presented, the accuracy numbers may measure the ability to predict Reddit popularity rather than humor. The paper's honest acknowledgment of the genre confound in transfer experiments (Section 6) is a strength that should be incorporated into the central claim rather than treated as an aside.","major_comments":[{"comment":"The binary label in the Reddit experiment is defined by a 200-upvote cutoff that was chosen after inspecting the score distribution ('We used this natural divide as the cutoff'). Because scores were collected hourly during March-April 2019 and later updated, the final score conflates humor with exposure time, posting time, and subreddit visibility, and the cutoff itself is never validated against an independent humor judgment. As a result, the 72.4% test accuracy in Table 2 may be measuring the model's ability to predict whether a post crossed an arbitrary popularity threshold rather than whether it is humorous. A concrete fix is to evaluate threshold sensitivity (e.g., report accuracy at several cutoffs) and to validate a subsample of the labels with independent human raters.","section":"Section 3.1"},{"comment":"The experiments are based on a single 75/25 stratified split with no confidence intervals, no multiple random seeds, and no significance tests. The headline comparison in Table 2 ('the Transformer achieved an accuracy of 72.4 percent ... while the CNN was in the high 60's') could therefore be within run-to-run variance. For a claim of this strength, repeated resampling or at least a significance test is necessary.","section":"Section 4.2 and Section 5.2"},{"comment":"The transfer experiments compare jokes against non-joke sentences drawn from news text. The near-ceiling F1 scores (93.1% and 98.6%) may be driven by an easy joke-versus-news genre distinction, a limitation the paper itself acknowledges in Section 6 ('it seems to be an easier task to tell when something isn't a joke at all'). To support the claim that the model understands humor, the negative class should include non-humorous but joke-like text, or the analysis should report performance on a genre-balanced control.","section":"Sections 3.2, 3.3 and Tables 3-4"},{"comment":"The human baseline in Table 2 is reported as '66.3%' with no formal definition. From Section 5.1 it appears this is the proportion of jokes for which the majority of Mechanical Turk raters labeled the joke humorous, not an accuracy against the Reddit labels. If so, it is not directly comparable to the model accuracy and should be either recalculated as agreement with the Reddit-derived labels or clearly labeled as a human agreement rate.","section":"Table 2 and Section 5.1"}],"minor_comments":[{"comment":"Figure 1 is a generic reproduction of the Vaswani et al. transformer and does not show the fine-tuned BERT classification head; consider replacing it with a diagram of the actual model or removing it.","section":"Section 4.1"},{"comment":"The phrase 'the inception ... of the Attention mechanism' is awkward, and the citation to Bahdanau et al. (2015) does not obviously support the surrounding historical claim; consider rewording and correcting the reference context.","section":"Section 1"},{"comment":"The paper does not report the posting dates of the collected jokes or a minimum age for posts entering the dataset; this information is needed to evaluate the exposure-time confound discussed in the major comments.","section":"Section 3.1"},{"comment":"The paper states that the Short Jokes negative set was built using 'the same method and news crawl source' as Chen and Soo (2018), but the exact sentence-matching criteria are not specified; please provide the matching details or a pointer to code.","section":"Section 3.2"},{"comment":"There are several typographical errors in the reference list (e.g., 'V on-Wun Soo' and 'Proceedings of the IEEE ,'), which should be corrected in a revision.","section":"References"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the Reddit r/Jokes dataset is a genuine addition, and the BERT baseline is competently run, but the label definition and the transfer evaluation undercut the 'humor detection' claim. Worth engaging with, but not as-is.\n\nWhat's new: the ~16k instance dataset with body/punchline splits, the MTurk comparison, and the zero-shot application of a Reddit-trained model to puns and short jokes. The dataset is open source, and the body/punchline distinction is a nice touch that lets them show the punchline carries more weight.\n\nThe problems are in the label and the comparisons. The 200-upvote cutoff is chosen after inspecting the score distribution. That's post hoc. And the scores were collected hourly over two months; a joke posted late in the window simply has less time to cross the threshold. So the label bundles humor together with exposure time and subreddit popularity. The paper would be much stronger if they validated the cutoff against independent human ratings or at least showed results across a range of thresholds. The absence of any confidence intervals or multiple seeds on the main split is also a real weakness for a result that rests on a 3-point accuracy gap.\n\nThe transfer experiments are cleaner but still not clean. The negative examples come from news text, so the model learns joke-vs-news style, not humor-vs-non-humor. The paper says this in the discussion ('easier task to tell when something isn't a joke at all'), but that sentence reads as a caveat, not as a fundamental qualifier on the F1 numbers. And the comparison is against previous CNN results trained on those datasets, not a same-protocol CNN baseline transferred from Reddit, so you can't separate the effect of BERT from the effect of the training data.\n\nThe citation pattern is fine—they cover the relevant humor detection literature and are transparent about which parts of the pipeline are new.\n\nWho's this for: anyone working on humor recognition or on using community ratings as supervision for subjective tasks. The dataset alone is worth having. But the central claim—that this is humor detection—needs tighter validation. I'd send it to a referee, but expect major revisions: threshold analysis, confidence intervals, and a same-setup CNN transfer baseline.","headline":"The Reddit r/Jokes dataset is a real resource, but the 200-upvote threshold and genre-confounded transfer experiments mean the paper shows popularity prediction and joke-style detection more than humor understanding.","tokens_in":6438,"tokens_out":3126,"would_cite":true,"duration_ms":29759,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a Transformer on Reddit upvote labels yields a humor detector that outperforms CNN baselines and general human raters on held-out jokes, and transfers to pun and short-joke classification.","keywords":["humor detection","BERT","Transformer","Reddit r/Jokes","joke classification","upvote labels","transfer learning","self-attention"],"falsifier":"Take held-out jokes whose upvote scores sit just below and just above 200, have a sample of r/Jokes readers rate them for humor, and compare those ratings with the 200-threshold labels; if human humor ratings do not track the threshold, or if the model's accuracy on those near-threshold jokes is at chance, the central claim fails.","tokens_in":5489,"feed_emoji":"😂","tokens_out":6627,"duration_ms":57068,"temperature":0.7,"pith_summary":"The paper claims that a machine can learn what makes a given online community laugh without any direct human annotation: jokes from Reddit's r/Jokes forum are labeled funny or not by whether their upvote score crosses 200, and a fine-tuned Transformer (BERT) trained on about 16,000 such labeled jokes predicts held-out jokes with 72.4% accuracy. That beats a CNN baseline, which lands in the high 60s, and general human raters, who agreed with the Reddit labels only 66.3% of the time. The same model, without further fine-tuning, identifies puns with an F1 of 93.1% and short jokes with 98.6%, surpassing previously reported CNN-based results. If true, this means humor detection can be reframed as a transferable text-classification task driven by community engagement signals rather than expert annotation.","feed_headline":"Transformer model spots funny Reddit jokes at 72.4%","feed_subtitle":"Fine-tuned BERT also beats CNN baselines on puns and short jokes","key_machinery":"The load-bearing mechanism is the Transformer's multi-head self-attention, instantiated in the pre-trained BERT encoder and fine-tuned with a binary classification layer. Its role is to weight words and sentence context so that the model distinguishes high-upvote from low-upvote r/Jokes posts; the training signal is supplied by Reddit's upvote distribution, with a score of 200 as the humor threshold, and by the body/punchline split of each post.","core_discovery":"The central discovery is that the self-attention of a pre-trained Transformer, fine-tuned on community upvote labels, captures enough about joke structure to predict what r/Jokes users will find funny. In experiments with the body alone, the punchline alone, and the full joke, the model reaches 72.4% accuracy on the full held-out set, compared with roughly 68% for the CNN baseline and 66.3% for general human raters. The paper also reports that the punchline contributes more to the prediction than the setup, and that the model transfers to two existing joke-identification benchmarks: Pun of the Day (F1 93.1%) and the Short Jokes dataset (F1 98.6%).","pith_inferences":["Upvotes are a continuous signal, so treating them as a binary above-or-below 200 discards information; a regression or ordinal model on the raw score may distinguish viral from merely funny jokes more sharply.","The human baseline came from general raters, not from r/Jokes readers; an accuracy comparison against the target community itself would likely be lower, since the model is tuned to the community's taste.","The large gap between the Transformer's accuracy on Reddit jokes (72.4%) and on short jokes (98.6%) suggests the Reddit task is the harder one, likely testing degree of humor within jokes rather than merely whether text is a joke at all."],"forward_implications":["A binary humor classifier can be built for any community that leaves numeric reactions, with no hand-labeled jokes required.","The same model trained on Reddit jokes transfers to pun and short-joke detection without retraining, so joke-likeness learned from one genre helps in another.","Because the punchline carries more predictive weight than the setup, humor-detection systems may improve by architecturally emphasizing the final segment of a joke.","Reported accuracy depends on a community-specific label, so model scores should be read as predictions of r/Jokes taste, not of universal humor."],"supporting_citations":[{"why":"Supplies the CNN baseline, the Short Jokes task setup, and the previous results the paper compares against.","marker":"Chen and Soo (2018)"},{"why":"Provides the pre-trained BERT Transformer that is fine-tuned as the humor classifier.","marker":"Devlin et al. (2018)"},{"why":"Defines the Transformer architecture and self-attention mechanism the model is built on.","marker":"Vaswani et al. (2017)"},{"why":"Created the Pun of the Day dataset and the previous humor-anchor method used as a comparison.","marker":"Yang et al. (2015)"},{"why":"Defines the Highway Networks used in the CNN baseline.","marker":"Srivastava et al. (2015)"}],"fun_headline_variants":["Transformer predicts Reddit jokes with 72.4% accuracy","AI model laughs at Reddit jokes: 72.4%","Punchline power: Transformer beats humans at joke detection","Self-attention gets the joke: 72.4% on r/Jokes","From Reddit to puns: Transformer excels at humor detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The binary label 'funny' is defined by a Reddit upvote score crossing 200; if upvotes reflect visibility, timing, or community in-crowd effects rather than humor, then the model is predicting an arbitrary score cutoff, not humor.","fun_headline_variants_meta":{"raw":{"variants":["Transformer predicts Reddit jokes with 72.4% accuracy","AI model laughs at Reddit jokes: 72.4%","Punchline power: Transformer beats humans at joke detection","Self-attention gets the joke: 72.4% on r/Jokes","From Reddit to puns: Transformer excels at humor detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000823,"raw_usage":{"total_tokens":3549,"prompt_tokens":844,"completion_tokens":2705,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":460,"completion_tokens_details":{"reasoning_tokens":2615}},"tokens_in":460,"tokens_out":2705,"duration_ms":73496,"temperature":1.0,"reasoning_tokens":2615,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:56:44.641873+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take held-out jokes whose upvote scores sit just below and just above 200, have a sample of r/Jokes readers rate them for humor, and compare those ratings with the 200-threshold labels; if human humor ratings do not track the threshold, or if the model's accuracy on those near-threshold jokes is at chance, the central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CNN baseline, the Short Jokes task setup, and the previous results the paper compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the pre-trained BERT Transformer that is fine-tuned as the humor classifier."},{"cited_title":"Gomez , Lukasz Kaiser , and Illia Polosukhin","cited_arxiv_id":null,"evidence_quote":"Defines the Transformer architecture and self-attention mechanism the model is built on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Created the Pun of the Day dataset and the previous humor-anchor method used as a comparison."}],"review_version":1}