Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Which LLMs Get the Joke? Probing Non-STEM Reasoning Abilities with HumorBench

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that LLM progress on STEM reasoning transfers to humor comprehension, and introduces HumorBench, an expert-annotated benchmark of cartoon-caption humor that remains unsolved on its hard subset.

desk verdict A genuinely new humor-comprehension benchmark with an honest validation story, but the transfer-to-STEM-reasoning claim leans on a post-hoc correlation and a confounded within-family comparison that the current length control does not fix. read the letter →

arxiv 2507.21476 v1 pith:NL2LCGWD submitted 2025-07-29 cs.CL cs.AI

Reuben Narad , Siddharth Suresh , Jiayi Chen , Pine S.L. Dysart-Bricken , Bob Mankoff , Robert Nowak , Jifan Zhang , Lalit Jain This is my paper
classification cs.CLcs.AI
keywords humorcomprehensionLLMbenchmarkingreasoningtransfernon-STEMLLM-as-judgeNewYorkerCaptionContesttest-timescalingobjectivejokeelements
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

HumorBench asks language models to explain, in their own words, what makes a cartoon-caption pair funny. Each joke is distilled into one to three hand-written "elements"—objective facts any correct explanation must state—and a GPT-4o autograder checks whether a model's explanation covers them. The paper's central claim is that performance on this task tracks performance on STEM reasoning benchmarks: HumorBench accuracy correlates positively with GPQA-Diamond, ARC-AGI, and LM Arena ELO, and after excluding ARC-tuned o-series models the ARC-AGI correlation reaches $\rho = 0.943$. Models trained only on STEM reasoning data, such as DeepSeek R1-Zero and Phi-4 Reasoning Plus, beat their base versions, which the paper reads as evidence that abstract reasoning transfers from STEM to humor. On the 100-element hard subset no current model exceeds 60% accuracy, so the benchmark is both sensitive to reasoning ability and still unsolved.

What carries the argument

The element annotation rubric is the load-bearing mechanism: each cartoon-caption pair is distilled into one to three concise, objectively verifiable statements that any correct explanation must include, such as "to the sharks, the person is the groceries." A GPT-4o autograder reads a model's free-text explanation and passes or fails each element against the rubric. This converts the open-ended task of humor explanation into an automatically gradable check, and it is what makes the benchmark's scores and cross-benchmark correlations interpretable.

What would settle it

Have a second team of experts independently annotate the same cartoon-caption pairs; if the two rubrics disagree often enough to change pass/fail outcomes on a substantial share of items, or if a model with near-zero scores on GPQA and ARC-AGI reaches the top of HumorBench, the transfer claim would collapse.

Watch

Extended reading notes

Core claim

The paper argues that humor comprehension, framed as explaining the objective elements of a joke, is largely a reasoning task. Scores on HumorBench correlate positively with established reasoning benchmarks: GPQA Diamond, ARC-AGI, and LM Arena ELO, with the ARC-AGI correlation reaching $\rho = 0.943$ once ARC-tuned o-series models are excluded. Models trained exclusively on STEM reasoning data—DeepSeek R1-Zero and Phi-4 Reasoning Plus—outperform their base versions on humor, suggesting that reasoning skills acquired in mathematics and coding transfer to non-STEM domains. The autograder agrees with human experts on 92% of 300 checked judgments but shows a leniency bias, so the authors state that reported HumorBench scores should be treated as upper bounds. On the hard subset of 100 elements, the best model, o3, passes only 59.85%, which the paper presents as evidence that HumorBench remains a sensitive, unsolved probe of higher-level reasoning.

Load-bearing premise

The benchmark's results stand on the assumption that the hand-written element rubrics are objective, complete ground truth for each joke and that an LLM judge checking an explanation against those elements captures whether a model really got the joke.

Editorial extensions

If this is right

  • HumorBench can serve as a non-STEM reasoning probe: under the paper's transfer claim, models that improve on STEM benchmarks should also improve here.
  • Reasoning-specific training, even when restricted to math and coding, transfers to humor comprehension, as shown by R1-Zero and Phi-4 Reasoning Plus beating their base versions.
  • Reported HumorBench scores should be treated as upper bounds, because the autograder's false-positive rate is roughly double its false-negative rate.
  • The 100-element hard subset remains unsolved: the best model, o3, passes only 59.85% of elements.
  • Increasing test-time compute does not produce uniform gains in humor reasoning; Claude 3.7 Sonnet gets worse when its thinking budget rises beyond 1024 tokens.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: if the transfer correlation is causal, HumorBench-style objective-element rubrics could be used as a training signal to keep general reasoning intact while models specialize in STEM; the paper does not test this use.
  • Editorial extension: the hard-subset examples that all models miss, such as the sharks treating the swimmer as groceries, point to a specific difficulty with adopting non-human or counterfactual perspectives; this is testable by making the required perspective more explicit in the caption and checking whether pass rates rise.
  • Editorial extension: the inverse test-time scaling seen in Claude 3.7 Sonnet may reflect overthinking rather than a fixed ceiling; comparing forced short versus long reasoning budgets on the same jokes would separate the two explanations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. HumorBench evaluates LLMs' ability to explain cartoon-caption humor against 499 hand-annotated "objective elements" covering roughly 300 New Yorker and Cartoonstock cartoon-caption pairs. A GPT-4o autograder checks whether a model's explanation explicitly covers each element; it is validated at 92% agreement with humans on a 300-judgment set. The paper reports that o3 leads the main set at 87.5% accuracy, that no model exceeds 60% on the 100-element HumorBench-Hard subset, that HumorBench scores correlate positively with GPQA-Diamond, ARC-AGI, and LM Arena ELO, that STEM-only reasoning models (R1-Zero, Phi-4 Reasoning Plus) beat their base versions, and that test-time scaling gives mixed results. From these findings the authors conclude that STEM reasoning transfers to humor comprehension and that HumorBench is a sensitive, still-unsolved probe of non-STEM reasoning.

Significance. If the transfer claim holds, HumorBench would be a valuable and badly needed non-STEM probe of reasoning, and the paper has several genuine strengths: the element-rubric design is concrete, the dataset went through an iterative refinement loop with inconsistency flags, a former New Yorker cartoon editor checked a random subset of 30 annotations, the autograder validation is explicit about its 92% agreement and its leniency bias, and the hard-subset analysis is honest about persistent model failures. The paper also deserves credit for stating in the Limitations section that scores should be treated as upper bounds. The main empirical claims, however, rest on two pillars that are currently weaker than the text suggests: cross-benchmark correlations over a small number of model-level points with a post-hoc exclusion, and within-family comparisons that are confounded by output format. The significance of the benchmark itself is not in doubt; the significance of the "STEM transfers to humor" conclusion depends on resolving those confounds.

major comments (3)
  1. [§4.2, Figure 6; §3.5 Length Control] The within-family comparisons that carry the transfer claim are not matched in output format. R1-Zero and Phi-4 Reasoning Plus include reasoning traces in their final responses, while DeepSeek-V3 and the base Phi-4 model do not; truncating all outputs to the last 1000 tokens (§3.5) does not strip reasoning traces or equate formats. Because the autograder prompt (Appendix B) passes an answer that "explicitly covers" the element, a verbose trace that mentions an element even once can yield PASS even if the model's stated conclusion is wrong. This systematically favors trace-emitting models and could produce the R1-Zero > V3 and Phi-4 Reasoning Plus > base gaps without any difference in humor comprehension. The paper acknowledges the traces in §4.2 but asserts that the length control makes the comparison fair, a claim that is not supported. Please re-run these two comparisons after removing or masking reasoning traces, or with a prompt that asks for the final explanation only, and report both versions together with the length-controlled numbers.
  2. [Table 2, §4.2] The cross-benchmark evidence for "transfer" is a small-N, partially post-hoc correlation. The headline ARC-AGI ρ=0.943 is reported only after excluding o-series models, leaving roughly ten model-level points; with such a small N, a single model can change the Spearman coefficient substantially, and excluding variants because their reported scores come from ARC-tuned checkpoints is a selection rule that should be justified rather than applied after seeing the data. In addition, positive rank correlations with GPQA, ARC-AGI, and LM Arena are equally consistent with a shared general-capability factor and do not by themselves establish that STEM progress transfers to humor. Please report the correlations with all models included, provide confidence intervals or permutation tests for ρ, and soften the causal "transfer" wording accordingly.
  3. [§3.3-3.4, Limitations] The benchmark's validity rests on the premise that the hand-written elements are objective, complete, and consistent ground truth for each joke. The validation is small relative to the dataset: a former editor reviewed only 30 annotations, the 300 autograder judgments cover explanations from only three models, and no inter-annotator agreement is reported for the element annotation process itself. The Limitations section also concedes that "some element annotations may still contain errors or reflect subjective interpretations," which is exactly the noise path that all reported accuracies and correlations inherit. Please report double-annotation agreement on a sample, describe how many annotators produced the elements and how disagreements were resolved, and provide a sensitivity analysis showing whether the headline rankings and the Table 2 correlations are stable under plausible element-level perturbations.
minor comments (5)
  1. [§3.2, §3.4] Please reconcile "approximately 300 unique cartoon-caption pairs" with "499 unique element annotations" by reporting the exact number of pairs and the distribution of elements per pair, since the dataset size is a stated limitation.
  2. [General] The dataset and autograder prompts are not linked in the manuscript; to make HumorBench usable and reproducible, the paper should release the annotations, cartoon descriptions, and judge prompts, or state clearly where they will be made available.
  3. [References] The reference list contains several integrity and formatting errors: the ToRA entry has a garbled author list, the Phi-4-reasoning technical report appears twice with the same title, and multiple entries use placeholder fragments such as "and 1 others." These should be corrected before publication.
  4. [§4.1, Figure 3] The main results figure is not reproduced with clear per-model labels in the text, and Section 4.1 does not report confidence intervals or per-model sample sizes for the accuracy numbers; because element outcomes are binary, binomial confidence intervals would help readers judge the significance of the reported gaps.
  5. [§4.4, Table 3] The humor-category annotations in Table 3 are produced by an o3-based categorization pipeline with no validation; a small human agreement check on a sample of elements would make the category-difficulty conclusions in Section 4.4 more trustworthy.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: HumorBench's claims are empirical, externally benchmarked, and not derived from fitted parameters or self-referential definitions.

full rationale

HumorBench is an empirical benchmark paper rather than a derivation from first principles. The central claims—STEM reasoning transfers to humor, STEM-only trained models perform well, and test-time scaling is mixed—are supported by measured accuracies and by correlations with external benchmarks (GPQA, ARC-AGI, LM Arena) that are not defined in terms of HumorBench scores. The element rubrics are hand-annotated under stated guidelines (Section 3.3) and refined through an iterative consistency process (Section 3.4), then independently checked against 300 human autograder judgments with 92% agreement (Table 1) and 30 expert-reviewed annotations; no parameter is fitted to HumorBench outcomes and then reported as a prediction. The closest issues are validity concerns, not circular derivation: (i) GPT-4o is both the autograder and an evaluated explainer, but the paper's own validation shows no GPT-4o explanation advantage and the Limitations section discloses the autograder is mildly lenient; (ii) HumorBench-hard is defined as the 100 elements most often missed, so reporting that models fail on it is partly a restatement of the selection criterion, but Section 4.4 explicitly presents this as a post hoc targeted analysis, not as a predicted result; (iii) self-citations (Jain et al. 2020; Zhang et al. 2024; Zhou et al. 2025) are used for data sources and motivation, and the conceptual point about objective versus subjective humor is not the load-bearing step in any empirical derivation. Per the hard rules, self-citation without a load-bearing reduction is not circularity, and no metric or equation in the paper is defined in terms of the target claims. The benchmark is self-contained against external benchmarks, so the appropriate finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The benchmark is an empirical measurement, so the ledger lists design assumptions rather than mathematical axioms. The hand-chosen thresholds are not fitted to HumorBench accuracy, so they are listed as procedural free parameters. The core assumptions are the objectivity of element labels, the validity of the GPT-4o judge, the sufficiency of textual descriptions, and the causal reading of correlation as transfer.

free parameters (2)
  • Annotation inconsistency flag threshold = 30%
    Chosen in Section 3.4 to flag element annotations for review; a higher or lower threshold changes which annotations remain in the 499-element set, but the value is not fitted to model scores.
  • Output truncation length = last 1000 tokens
    Chosen in Section 3.5 to standardize long reasoning traces; this is a fairness choice, not a parameter learned from HumorBench data.
assumptions (4)
  • domain assumption Element annotations are objective ground truth.
    Section 3.3 defines elements as objective facts; benchmark scores are computed against these annotations.
  • domain assumption GPT-4o autograder approximates human judgment for PASS/FAIL.
    Section 3.5 reports 92 percent agreement on 300 human-labeled examples, but all evaluations use this model-based judge.
  • domain assumption Textual descriptions retain all joke-critical visual information.
    Section 3.2 says visual style is omitted; if a necessary visual cue is absent, explanations cannot succeed.
  • domain assumption STEM benchmark correlations and paired STEM-trained model improvements indicate transfer of reasoning ability.
    Section 4.2 interprets correlation and two paired gains as transfer; alternative explanations include general capability scaling.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Which LLMs Get the Joke? Probing Non-STEM Reasoning Abilities with HumorBench." pith.science (2026). https://pith.science/paper/NL2LCGWD

@misc{pith2026250721476,
  author       = {Pith},
  title        = {Pith review of: Which LLMs Get the Joke? Probing Non-STEM Reasoning Abilities with HumorBench},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NL2LCGWD}},
  note         = {Machine review of arXiv:2507.21476}
}
read the original abstract

We present HumorBench, a benchmark designed to evaluate large language models' (LLMs) ability to reason about and explain sophisticated humor in cartoon captions. As reasoning models increasingly saturate existing benchmarks in mathematics and science, novel and challenging evaluations of model intelligence beyond STEM domains are essential. Reasoning is fundamentally involved in text-based humor comprehension, requiring the identification of connections between concepts in cartoons/captions and external cultural references, wordplays, and other mechanisms. HumorBench includes approximately 300 unique cartoon-caption pairs from the New Yorker Caption Contest and Cartoonstock.com, with expert-annotated evaluation rubrics identifying essential joke elements. LLMs are evaluated based on their explanations towards the humor and abilities in identifying the joke elements. To perform well on this task, models must form and test hypotheses about associations between concepts, potentially backtracking from initial interpretations to arrive at the most plausible explanation. Our extensive benchmarking of current SOTA models reveals three key insights: (1) LLM progress on STEM reasoning transfers effectively to humor comprehension; (2) models trained exclusively on STEM reasoning data still perform well on HumorBench, demonstrating strong transferability of reasoning abilities; and (3) test-time scaling by increasing thinking token budgets yields mixed results across different models in humor reasoning.

Figures

Figures reproduced from arXiv: 2507.21476 by the authors.

Figure 1
Figure 1. Overview of our humor analysis approach. (a) We distinguish between [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Example HUMORBENCH annotation. The cartoon (left) is paired with its description, caption, and two hand-labeled joke elements (right). 4 Experiments Along with creating the HumorBench evaluation, we extensively benchmarked current frontier mod￾els. For consistency, all models are given the same prompt and scaffolding describing the task (see Ap￾pendix A). We arrived at this prompt after validat￾ing across several di… view at source ↗
Figure 3
Figure 3. Benchmarking results on frontier models In general, the results from the main benchmark￾ing effort were unsurprising. As shown in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: HumorBench performance compared to several common benchmarks. We see positive correlation with [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 6
Figure 6. Figure 6: Deepseek R1 Zero and Phi-4 Reasoning Plus, [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 5
Figure 5. Figure 5: HumorBench test-time compute experiments. Note, "mean output tokens" includes both reasoning and [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 7
Figure 7. Figure 7: Token usage with different ‘thinking budget‘ [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Examples of elements in the HUMORBENCH hard subset. See [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HumorRank: A Tournament-Based Leaderboard for Evaluating Humor Generation in Large Language Models

    cs.CL 2026-03 unverdicted novelty 7.0 of 10

    HumorRank ranks nine LLMs on textual humor using GTVH-grounded pairwise tournaments and Adaptive Swiss aggregation on the SemEval-2026 MWAHAHA dataset, finding that comedic mechanism mastery matters more than scale.

  2. Relationship-Centered Care: Relatedness and Responsible Design for Human Connections in Mental-Health Care

    cs.HC 2026-03 accept novelty 7.0 of 10

    Length-aware GFlowNet matching of α-power base distributions elicits either stronger LLM reasoning (α>1) or restored creativity (α<1) without external supervision, matching or beating RLIF and GRPO baselines.

Reference graph

Works this paper leans on

27 extracted references · 9 canonical work pages · cited by 2 Pith papers

  1. [1]

    Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mo- jan Javaheripi, Neel Joshi, Piero Kauffmann, Yash Lara, Caio C. T. Mendes, Arindam Mitra, Besmira Nushi, Dimitris Papailiopoulos, Olli Saarikivi, Shital Shah, Vaishnavi Shrivastava, and 4 others. 2025a. Phi-4-reasoning t...

    arXiv

  2. [4]

    https: //blog.google/technology/google-deepmind/ gemini-model-thinking-updates-march-2025/

    Gemini 2.5 pro: Our most intelligent ai model. https: //blog.google/technology/google-deepmind/ gemini-model-thinking-updates-march-2025/ . Accessed 19 May

    work page 2025

  3. [5]

    In Working Notes of CLEF 2025 - Conference and Labs of the Evaluation F orum

    Overview of the clef 2025 joker lab: Automatic hu- mour processing–recognition, retrieval, and genera- tion. In Working Notes of CLEF 2025 - Conference and Labs of the Evaluation F orum. Google

    work page 2025

  4. [7]

    arXiv preprint arXiv:2502.19187

    Big-bench extra hard. arXiv preprint arXiv:2502.19187. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra

    arXiv

  5. [9]

    When 'YES' Meets 'BUT': Can Large Models Comprehend Contradictory Humor Through Comparative Reasoning?

    When ’yes’ meets ’but’: Can large models comprehend contradictory 9 humor through comparative reasoning? Preprint, arXiv:2503.23137. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Tom Brown, Ben Mann, Sheer Agnihotri, Casey Möller, Nick Kalaitzis, and 1 others

    work page Pith review arXiv

  6. [10]

    arXiv preprint arXiv:2305.20050

    Let’s verify step by step. arXiv preprint arXiv:2305.20050. Yang Liu, Dan Iter, Yichong Xu, and Shuohang Wang Yelong Peng

    arXiv

  7. [11]

    In Proceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11069–11081

    G-eval: Nlg evaluation using gpt-4 with better human alignment. In Proceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11069–11081. Association for Computational Linguistics. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian- guang Lou, Chongyang Tao, Xiubo Geng, Qing- wei Lin, Shifeng Chen, Yansong Tang, and Don...

    work page 2023

  8. [12]

    arXiv preprint arXiv:2308.09583

    Wizardmath: Empowering mathematical reasoning for large language mod- els via reinforced evol-instruct. arXiv preprint arXiv:2308.09583. Ian R McKenzie, Alexander Lyzhov, Michael Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Aaron Kirtland, Alexis Ross, Alisa Liu, and 1 others

    arXiv

Show all 27 references
  1. [13]

    arXiv preprint arXiv:2306.09479

    Inverse scaling: When bigger isn’t better. arXiv preprint arXiv:2306.09479. Meta

    arXiv

  2. [15]

    arXiv preprint arXiv:2501.01257

    Codeelo: Bench- marking competition-level code generation of llms with human-comparable elo ratings. arXiv preprint arXiv:2501.01257. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R. Bowman

    arXiv

  3. [16]

    arXiv preprint arXiv:2311.12022

    Gpqa: A graduate-level google-proof q&a bench- mark. arXiv preprint arXiv:2311.12022. Zhihong Shao, Ganqu Lu, Jishuai Yu, Hang Wang, Cong Zheng, Yu Wang, Tao Zhao, Zherui Yang, Xipeng Chen, Xuetao Mao, Dan Su, and Zhilin Xu

    arXiv

  4. [17]

    arXiv preprint arXiv:2402.03300

    Deepseek-math: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou

    arXiv

  5. [18]

    arXiv preprint arXiv:2501.12077

    Paperbench: Bench- marking large language models for scientific paper understanding and knowledge production. arXiv preprint arXiv:2501.12077. Jinyan Su, Jennifer Healey, Preslav Nakov, and Claire Cardie

    arXiv

  6. [19]

    arXiv preprint arXiv:2505.00127

    Between underthinking and overthink- ing: An empirical study of reasoning length and cor- rectness in llms. arXiv preprint arXiv:2505.00127. Haoxiang Sun, Yingqian Min, Zhipeng Chen, Wayne Xin Zhao, Zheng Liu, Zhongyuan Wang, Lei Fang, and Ji-Rong Wen

    arXiv

  7. [20]

    arXiv preprint arXiv:2503.21380

    Challenging the boundaries of reasoning: An olympiad-level math benchmark for large language models. arXiv preprint arXiv:2503.21380. Jiao Sun, Ajay Nagesh, and Zachary C. Lipton

    arXiv

  8. [21]

    In Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Processing, pages 2866–2879

    Expunations: Augmenting puns with keywords and explanations. In Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Processing, pages 2866–2879. Association for Com- putational Linguistics. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and...

    2022

  9. [22]

    arXiv preprint arXiv:2401.02642

    Emergent reasoning in large language models. arXiv preprint arXiv:2401.02642. Zixuan Xu, He He, Emmanuele Chersoni, Yi-Lin Tuan, and Qin Lu

    arXiv

  10. [23]

    In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 4213–4228

    A good pun is its own reword: Evaluating pun recognition, explanation, and genera- tion in llms. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 4213–4228. Association fo...

    2024

  11. [24]

    arXiv preprint arXiv:2401.14273

    Mathglm: Towards generalizable mathematical rea- soning via language modeling. arXiv preprint arXiv:2401.14273. Wenkai Yang, Shuming Ma, Yankai Lin, and Furu Wei

    arXiv

  12. [25]

    arXiv preprint arXiv:2502.18080

    Towards thinking-optimal scaling of test-time compute for llm reasoning. arXiv preprint arXiv:2502.18080. Xiang Yue, Boshi Ma, Yifan Wang, Junhong Chen, Xu- anhe Gu, and Diyi Liang

  13. [26]

    arXiv preprint arXiv:2406.10522

    Humor in ai: Massive scale crowd-sourced preferences and benchmarks for cartoon captioning. arXiv preprint arXiv:2406.10522. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Joseph E. Gonzalez, Ion Stoica, a...

    arXiv

  14. [27]

    arXiv preprint arXiv:2502.20356

    Bridg- ing the creativity understanding gap: Small-scale human alignment enables expert-level humor ranking in llms. arXiv preprint arXiv:2502.20356. 11 A Main Benchmark Prompt You are a humor expert extraordinaire, judging the New Yorker Cartoon Caption Contest. Your current ...

    arXiv

  15. [2020]

    arXiv preprint arXiv:2004.10645

    Ambigqa: Answering ambiguous open-domain questions. arXiv preprint arXiv:2004.10645. Published as a long paper at EMNLP

    2004 arXiv

  16. [2022]

    arXiv preprint arXiv:2206.14858

    Solving quantita- tive reasoning problems with language models. arXiv preprint arXiv:2206.14858. Tuo Liang, Zhe Hu, Jing Li, Hao Zhang, Yiren Lu, Yunlai Zhou, Yiran Qiao, Disheng Liu, Jeirui Peng, Jing Ma, and Yu Yin

    arXiv

  17. [2023]

    arXiv preprint arXiv:2309.17452

    Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint arXiv:2309.17452. Jack Hessel, Ana Marasovi ´c, Jena D. Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff, and Yejin Choi

    arXiv

  18. [2024]

    arXiv preprint arXiv:2403.04132

    Chatbot arena: An open platform for evaluating llms by hu- man preference. arXiv preprint arXiv:2403.04132. Francois Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers

    arXiv

  19. [2025]

    arXiv preprint arXiv:2412.04604

    Arc prize 2024: Technical report. arXiv preprint arXiv:2412.04604. Google DeepMind

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.