Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

CleanComedy: Creating Friendly Humor through Generative Techniques

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A multi-stage filtering pipeline cuts joke toxicity by about half and yields reusable English and Russian humor datasets with human scores, while two-stage fine-tuning produces cleaner yet less funny LLM humor.

desk verdict A genuinely useful bilingual humor dataset, honest about its limits, but the English evaluation rests on non-native annotators and the alignment claims are weaker than the paper suggests. read the letter →

arxiv 2412.09203 v1 pith:UA7NT2YQ submitted 2024-12-12 cs.CL

Dmitry Vikhorev , Daria Galimzianova , Svetlana Gorovaia , Elizaveta Zhemchuzhina , Ivan P. Yamshchikov This is my paper · ORCID
classification cs.CL
keywords computationalhumorgenerationtoxicityfilteringdatasetcurationhumanevaluationLLMalignmentEnglishandRussianjokes
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces CleanComedy, a toxicity-filtered corpus of English and Russian jokes built by merging existing joke collections and passing them through a multi-stage cleaning pipeline. The paper reports that the pipeline cuts the toxicity percentage in the final datasets to about half of the unfiltered source data, with 44,481 English and 40,926 Russian jokes surviving, and adds CleanComedy Gold, 1,000 jokes per language each rated by five annotators on a 1–5 humor scale. Using these datasets, the authors fine-tune an 8-billion-parameter language model with LoRA and a soft-label alignment stage that turns average human scores into training targets. In a human evaluation of 600 jokes, the curated human jokes and GPT-4o score highest on funniness, while the fine-tuned models produce cleaner but less funny output. The contribution, if correct, is a reusable lower-toxicity humor resource plus a reproducible training recipe for safer joke generation.

What carries the argument

The load-bearing mechanism is a cascaded filtering pipeline followed by a two-stage training procedure. The filter chain keeps only entries between 50 and 150 characters, removes toxic texts with Detoxify for English and ruBERTConv for Russian (scores above 0.1 are dropped), deduplicates semantically with Sentence-BERT cosine thresholds of 0.7 (English) and 0.9 (Russian), labels jokes with a zero-shot DeBERTa-v3 classifier to remove political, racist, and insulting content, and finally deletes whole BERTopic clusters on sensitive topics such as religion, funerals, bathrooms, officers, pregnancy, nations, disabilities, and divorce. On the modeling side, the paper fine-tunes an 8-billion-parameter language model (the base, non-instruction version of Llama 3.1) with LoRA at rank 4, first by supervised fine-tuning on the clean jokes, then by an alignment stage where average human scores are linearly mapped to soft labels and trained with binary cross-entropy, an approach inspired by DPO and SimPO but using soft labels instead of chosen-rejected pairs.

What would settle it

Re-annotate the CleanComedy Gold sets with a demographically diverse panel of native English and Russian speakers, then compare their funniness and offensiveness ratings with the pipeline's automated filter decisions; a weak correlation, or a toxicity gap much smaller than half under a different toxicity classifier, would contradict the central curation claim.

Watch

Extended reading notes

Core claim

The paper's central claim is that automated curation can produce a humor dataset that is roughly half as toxic as its raw sources, and that training on that curated data shifts a language model toward cleaner humor generation without matching the funniness of large general models or human-written jokes. The reported toxicity percentages for the filtered datasets are 11.41% (English) and 11.37% (Russian), compared with 26.09% and 20.93% for unfiltered samples; the human funniness scores place clean human jokes at 2.96 (English) and 2.84 (Russian), GPT-4o at 3.02 and 2.38, and the aligned fine-tuned model at 2.02 and 1.74. The authors take these numbers as evidence that the filtering pipeline works and that two-stage fine-tuning with soft labels can steer generation toward friendlier humor, while acknowledging that generative humor remains an open problem.

Load-bearing premise

The pipeline assumes that its automated proxies—toxicity classifier scores, SBERT similarity cutoffs, zero-shot content labels, and deleted BERTopic clusters—are a reliable stand-in for what a broad audience finds offensive; if those proxies are biased, the 'clean' dataset and the reported toxicity reduction inherit that bias.

Editorial extensions

If this is right

  • CleanComedy provides roughly 44,000 English and 41,000 Russian jokes with reduced toxicity and duplication, plus per-joke human scores for 1,000 jokes per language, as reusable training and evaluation material.
  • The two-stage LoRA plus soft-label alignment recipe offers a lightweight way to make an 8B model generate cleaner humor, which could transfer to other constrained creative-text tasks.
  • Toxicity filtering roughly halves the share of offensive jokes compared with the unfiltered source collections, making the pipeline a candidate template for other dataset curation efforts.
  • Because the filtered fine-tuned models score lower on funniness than GPT-4o and human jokes, the paper implies that safety filtering alone does not produce funnier jokes, so humor quality needs separate optimization.
  • Publishing individual annotator scores rather than only averages enables downstream research on humor personalization and on how demographic factors shape funniness ratings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Nothing in the paper tests whether the same filter cascade transfers to other languages or joke formats; a natural extension is to run the pipeline on a third language and check whether the toxicity-halving result persists.
  • The reported drop in funniness for filtered jokes suggests a possible toxicity–funniness tradeoff that the paper does not isolate; an experiment that varies the toxicity threshold and measures both metrics would make that tradeoff explicit.
  • Because the annotator pool skews young (20–30) and includes few native English speakers, the human scores likely reflect a specific demographic's humor; re-annotation with a broader panel would show how much the headline numbers depend on that panel.
  • The soft-label alignment loss could be adapted to other subjective attributes besides humor, such as politeness or helpfulness, where scalar human ratings are easier to collect than pairwise preference data.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces CleanComedy, a toxicity-filtered and deduplicated joke dataset in English and Russian, with human humor scores for 1,000 jokes per language (CleanComedy Gold). The curation pipeline combines exact and semantic deduplication, automated toxicity classifiers, zero-shot labeling, and BERTopic cluster removal. The authors then fine-tune Llama-3.1-8B on the clean data and apply an alignment stage that uses the Gold humor scores as soft labels. They evaluate the dataset and the models by sampling 100 jokes from each of six groups (SFT, SFT+Aligned, Instruct, GPT-4o, Unfiltered Dataset, Clean Dataset) and collecting human ratings of humor and toxicity. The central claim is that the clean dataset has about half the toxicity of the unfiltered data, and that the two-stage training yields ethically aligned humor generation. The paper releases the dataset and individual annotations on GitHub.

Significance. If the claims hold, CleanComedy is a useful resource: it provides a reproducible filtering pipeline, a public dataset in two languages, and per-joke human scores that can support personalization research. The explicit release of individual annotations rather than only aggregates is a strength. The paper is also commendably transparent about annotator demographics. However, the central quantitative claims are weakened by the annotator pool for English and by the absence of significance testing; the alignment conclusion is not supported by the reported table. These issues are fixable within the manuscript's scope, so the contribution is potentially valid but needs revision.

major comments (3)
  1. [Section 5, Table 2 and Figure 10] The English half of the central toxicity-halving claim is not established for the target population. The paper states that 'there was almost no native speakers of English among our volunteers (Figure 10), which has significantly influenced the humor scores obtained for the English jokes.' Because the same annotators answered the binary 'vulnerable or inappropriate?' question, the English toxicity drop from 26.09% to 11.41% and the Clean Dataset humor score of 2.96 are measurements of non-native perception, not of the English-speaking audience the dataset is intended to serve. The authors should either re-annotate a subsample with native English speakers, or explicitly restrict the claim to the actual annotator population and provide evidence that toxicity and humor judgments are stable across populations. This is an external-validity gap, not an internal inconsistency, but it is load-bearing for the paper's main quantitative contribution.
  2. [Section 5, Table 2] The toxicity percentages are not defined or tested. The text reports a percentage but does not state how individual 'yes/no' answers were aggregated into a joke-level or group-level toxicity label (e.g., any annotator, majority, or all annotators). In addition, no confidence intervals or significance tests are provided. Under a simple two-proportion z-test, the Russian drop from 20.93% to 11.37% is not significant at the 0.05 level (z ≈ 1.85, p ≈ 0.06), so the claimed 'half' reduction is not statistically supported for Russian. The English drop is significant under the same naive test, but the lack of a defined aggregation rule and the absence of annotation-clustering corrections make all point estimates difficult to interpret. The authors should report the aggregation rule, confidence intervals, and significance tests that account for annotator random effects.
  3. [Section 4.2, Eq. (1)-(2); Section 5 and 6] The claim that alignment improves generated humor is not supported by the reported data. The alignment stage trains the model to predict average humor scores from CleanComedy Gold, using the same five-point scale and annotation protocol as the evaluation in Section 5. Yet Table 2 shows that the SFT+Aligned model does not outperform the SFT-only model on humor scores in either language (English: 2.02 vs 2.11; Russian: 1.74 vs 1.68), and no significance tests are given. The conclusion that the results 'underscore the importance of alignment techniques in improving the quality and relevance of generated humor' is therefore overstated. The authors should either provide statistical evidence of improvement or temper the conclusion to reflect that alignment mainly reduced Russian toxicity (4.01% to 3.3%) in their sample.
minor comments (6)
  1. [Section 3.1] The text contains two unresolved cross-references to 'Table ?? in the Appendix'; the intended tables appear to be Tables 5 and 6 in the appendix, and the references should be fixed.
  2. [Section 3.2 and Section 5] The paper does not report inter-annotator agreement for the Gold annotations or for the evaluation (e.g., Krippendorff's alpha or ICC). Such measures would help readers interpret the reliability of the humor and toxicity scores.
  3. [Section 5] The definition of the toxicity percentage is ambiguous: it is not stated whether a joke is counted as toxic if at least one annotator marks it as vulnerable/inappropriate or only if a majority does. This should be clarified in the text or a footnote.
  4. [Section 5] The procedural difference for the English LLaMA 3.1 8B (Instruct) generation (temperature 0.9 and semantic deduplication) is mentioned in a footnote but not discussed in terms of comparability. The authors should report how many duplicates were removed and justify the asymmetry, or use a common protocol for both languages.
  5. [Section 3.1] The phrase 'unbiased toxicity classifier' is imprecise, since Detoxify is trained on a particular definition of toxicity and the authors later acknowledge classifier limitations. Consider rewording to avoid overclaiming objectivity.
  6. [Table 1] The CleanComedy English row lists a '2-scale score' while the Gold row lists a '5-scale score'; the '2-scale' appears to refer to the binary toxicity annotation, but this is confusing and should be labeled more explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the curation pipeline and human evaluation are externally grounded.

full rationale

The paper's central claims are (1) that CleanComedy is a toxicity-filtered, deduplicated humor dataset, and (2) that fine-tuned and aligned LLMs can generate humor comparable to human-written jokes. The filtering pipeline relies on external classifiers (Detoxify, ruBERTConv), a zero-shot DeBERTa model, SBERT similarity thresholds, and BERTopic cluster removal. The effectiveness of this filtering is then tested by human binary toxicity judgments on 100-sample unfiltered versus clean groups (Table 2). Because the filters are automated and the evaluation is human, the toxicity-halving claim is not a tautology and does not reduce to the filter outputs by construction. The alignment stage trains the LLM with soft labels derived from CleanComedy Gold mean humor scores, but the evaluated outputs are newly generated jokes rated by human annotators; the model never sees the evaluation jokes, so the humor scores in Table 2 are not the training targets. No equation in the paper is definitionally equal to another, no fitted parameter is renamed as a prediction, and no load-bearing argument rests on a self-citation. The acknowledged non-native English annotator bias (Section 5, Figure 10) is an external-validity limitation, not evidence of circularity. Accordingly, the derivation chain is self-contained against external benchmarks and human judgment, and no circular step is present.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central dataset claim rests on the validity of the classifier-based filtering thresholds and the representativeness of the volunteer annotator pool. The filtering thresholds are hand-picked from data inspection, which makes the toxicity reduction result sensitive to those choices.

free parameters (5)
  • cosine deduplication threshold = 0.7 English, 0.9 Russian
    Chosen after inspecting duplicate examples in Section 3.1; no validation that it preserves joke diversity.
  • length filter bounds = 50 to 150 characters (Section 3.1); Table 3 caption says 30 to 150
    Hand-chosen to remove short noise and long repetitions; inconsistent with the appendix.
  • ruBERTConv toxicity threshold = >0.1 considered toxic
    Set in Table 4 caption; arbitrary cutoff for Russian toxicity.
  • topic clusters removed = religion, funerals, bathrooms, officers, pregnancy, nations, disabilities, divorce
    Chosen after cluster inspection in Section 3.1; no external validation that these topics are universally inappropriate.
  • zero-shot label set = politics, neutral, offending, alcohol, drugs, racist
    Author-defined categories for DeBERTa filtering in Section 3.1.
assumptions (4)
  • domain assumption Detoxify, ruBERTConv, and DeBERTa classifier outputs correspond to human judgments of toxicity and offensiveness.
    Section 3.1 uses these models to delete content; the paper acknowledges Detoxify flags profanity regardless of tone.
  • domain assumption Average scores from five voluntary annotators are reliable gold labels for humor.
    Section 3.2; annotators are mostly 20-30 and almost no native English speakers, so the gold standard is demographic-specific.
  • ad hoc to paper Removing entire topic clusters (religion, funerals, bathrooms, pregnancy, etc.) improves ethical quality without disproportionately removing funny jokes.
    Section 3.1; no measurement of humor loss from these deletions.
  • domain assumption SBERT cosine similarity thresholds capture semantic duplication across languages.
    Section 3.1; thresholds differ by language and are not calibrated against human duplicate judgments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CleanComedy: Creating Friendly Humor through Generative Techniques." pith.science (2026). https://pith.science/paper/UA7NT2YQ

@misc{pith2026241209203,
  author       = {Pith},
  title        = {Pith review of: CleanComedy: Creating Friendly Humor through Generative Techniques},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UA7NT2YQ}},
  note         = {Machine review of arXiv:2412.09203}
}
read the original abstract

Humor generation is a challenging task in natural language processing due to limited resources and the quality of existing datasets. Available humor language resources often suffer from toxicity and duplication, limiting their effectiveness for training robust models. This paper proposes CleanComedy, a specialized, partially annotated toxicity-filtered corpus of English and Russian jokes collected from various sources. We study the effectiveness of our data filtering approach through a survey on humor and toxicity levels in various joke groups. In addition, we study advances in computer humor generation by comparing jokes written by humans with various groups of generative jokes, including our baseline models trained on the CleanComedy datasets.

Figures

Figures reproduced from arXiv: 2412.09203 by the authors.

Figure 1
Figure 1. Topic modelling for CleanComedy English. [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Topic modelling for CleanComedy Russian. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Average scores for CleanComedy Gold datasets in English (in the left picture) and Russian (in the right picture). The average score of 5 annotators was computed for each joke. 5 Results For both languages, we sample 100 examples from each of the following six groups: 1. LLaMA 3.1 8B (Supervised Fine-Tuned). See Supervised Fine-Tuning section for training details. 2. LLaMA 3.1 8B (Supervised Fine-Tuned + Aligned). Se… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Topic modeling for CleanComedy English [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Topic modeling for CleanComedy English. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Number of jokes rated by one person for English (in the left picture) and Russian (in the right picture). [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Age of annotators for English (in the left picture) and Russian (in the right picture). Only annotators with [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Gender of annotators for English (in the left picture) and Russian (in the right picture). Only annotators [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Education level of annotators for English (in the left picture) and Russian (in the right picture). Only [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Language level of annotators. Only annotators with at least one evaluation point are taken into account. [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HumorGen: Cognitive Synergy for Humor Generation in Large Language Models via Persona-Based Distillation

    cs.CL 2026-03 unverdicted novelty 6.0 of 10

    Persona-based Mixture-of-Thought data curation lets a 7B student outperform larger models on humor generation, while DPO and O-GRPO add no gain over SFT.

  2. On the Wings of Imagination: Conflicting Script-based Multi-role Framework for Humor Caption Generation

    cs.CL 2026-02 conditional novelty 6.0 of 10

    A GTVH-driven multi-role LLM framework with conflicting-script extraction and humor-retrieval imagination trees improves cartoon caption funniness over previous methods on New Yorker benchmarks.

Reference graph

Works this paper leans on

16 extracted references · 10 canonical work pages · cited by 2 Pith papers

  1. [2]

    Knowledge Amalgam: Generating Jokes and Quotes Together

    Knowledge amalgam: Generating jokes and quotes together. Preprint, arXiv:1806.04387. 8 Daryna Dementieva, Daniil Moskovskiy, Varvara Logacheva, David Dale, Olga Kozlova, Nikita Semenov, and Alexander Panchenko

    work page Pith review arXiv

  2. [5]

    Preprint, arXiv:2106.09685

    Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685. Sophie Jentzsch and Kristian Kersting

    arXiv

  3. [7]

    In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 105–119, Online

    SemEval 2021 task 7: HaHackathon, detecting and rating humor and offense. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 105–119, Online. Association for Computational Linguistics. Yu Meng, Mengzhou Xia, and Danqi Chen

    work page 2021

  4. [9]

    In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 1622–1636

    A robot walks into a bar: Can language models serve as creativity supporttools for comedy? an evaluation of llms’ humour alignment with comedians. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 1622–1636. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn

    work page 2024

  5. [10]

    Preprint, arXiv:2305.18290

    Direct preference optimization: Y our language model is secretly a reward model. Preprint, arXiv:2305.18290. Nils Reimers and Iryna Gurevych

    arXiv

  6. [11]

    In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing

    Sentence- bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. Nils Reimers and Iryna Gurevych

    work page 2019

  7. [13]

    The Naughtyformer: A Transformer Understands Offensive Humor

    The naughtyformer: A transformer understands offensive humor. Preprint, arXiv:2211.14369. Bolotova V . V ., Blinov V . A., Mishchenko K. I., and Braslavski P . I

    work page Pith review arXiv

  8. [15]

    Humor Detection: A Transformer Gets the Last Laugh

    Humor detection: A transformer gets the last laugh. Preprint, arXiv:1909.00252. Yuhan Wu et al

    work page Pith review arXiv 1909

Show all 16 references
  1. [2015]

    In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , pages 2367–2376, Lisbon, Portugal

    Humor recognition and humor anchor extraction. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , pages 2367–2376, Lisbon, Portugal. Association for Computational Linguistics. Wei Zhang et al

    2015

  2. [2017]

    Dialogue 2017

    Which ir model has a better sense of humor? search over a large collection of jokes. In Computational Linguistics and Intellectual Technologies: Proceedings of the International Conference “Dialogue 2017”, Moscow, Russia. Russian State University For The Humanities. Orion Well...

    2017

  3. [2018]

    Humor recognition using deep learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 113–117, New Orleans, Louisiana. Association for Computation...

    2018

  4. [2019]

    Preprint, arXiv:1905.07213

    Adaptation of deep bidirectional multilingual transformers for russian language. Preprint, arXiv:1905.07213. Moritz Laurer, Wouter Van Atteveldt, Andreu Casas, and Kasper Welbers

    1905 arXiv

  5. [2020]

    arXiv preprint arXiv:2004.09813

    Making monolingual sentence embeddings multilingual using knowledge distillation. arXiv preprint arXiv:2004.09813. Pradeep Sunkara et al

    2004 arXiv

  6. [2021]

    Preprint, arXiv:2105.09052

    Methods for detoxification of texts for the russian language. Preprint, arXiv:2105.09052. Rohan D’Silva

    arXiv

  7. [2022]

    arXiv preprint arXiv:2203.05794

    Bertopic: Neural topic modeling with a class-based tf-idf procedure. arXiv preprint arXiv:2203.05794. Laura Hanu and Unitary team

    arXiv

  8. [2024]

    Preprint, arXiv:2405.14734

    Simpo: Simple preference optimization with a reference-free reward. Preprint, arXiv:2405.14734. Rada Mihalcea and Carlo Strapparava

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.