Automatically Detecting Amusing Games in Wordle

Adam Kabbara; Cindy Liu; Gary Liang; Kina Kim; Michael Guerzhoy; Minahil Bakhtawar; Ronaldo Luo

REVIEW 4 major objections 6 minor 12 references

Automatically Detecting Amusing Games in Wordle

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that whether a Wordle game amuses Reddit users can be predicted weakly but above chance from game statistics alone, with a logistic regression reaching 54.5% correct classification against a 50% baseline.

desk verdict Weak but real signal for predicting GPT-3.5-labeled amusement in Wordle; the human-label validation is too thin to support the 'user amusement' framing. read the letter →

arxiv 2506.05415 v1 pith:AGXE4S6E submitted 2025-06-04 cs.CL

Ronaldo Luo , Gary Liang , Cindy Liu , Adam Kabbara , Minahil Bakhtawar , Kina Kim , Michael Guerzhoy This is my paper

classification cs.CL

keywords Wordleamusementpredictionhumordetectionfew-shotGPT-3.5classificationlogisticregressiongamefeaturescomputationalcreativityRedditcomments

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that user amusement at Wordle games can be predicted from game statistics, to a modest but real extent. The authors scrape about 80,000 Reddit reactions to Wordle games, label them as amused or not with a few-shot GPT-3.5 classifier, and verify that those labels coincide weakly with human annotations. They then train a logistic regression on aggregate game features—number of guesses, reductions in the number of possible answers, Levenshtein and GloVe distances between guesses, and predicted intrinsic funniness of guessed words—and reach 54.5% correct classification on a balanced test set of 15,000 games, against a 50% baseline. The authors read this as evidence that a measurable aspect of the humor and creativity infused into Wordle games lives in the game itself, not only in the comment threads.

What carries the argument

The central mechanism is a feature set computed from each Wordle game transcript. For every guess the authors compute the reduction in the number of words still consistent with the color feedback, the Levenshtein and GloVe distances to the previous guess, and a predicted 'intrinsic funniness' score for the guessed word; the funniness score comes from a linear regression trained on 4,858 words with human humor ratings using features such as category-defining vectors and valence-arousal-concreteness estimates. These aggregate statistics are fed into a logistic regression classifier whose coefficients reveal which game properties shift the predicted probability of an amused reaction, and the label side is a few-shot GPT-3.5 model that turns each Reddit comment into a binary amused/not-amused judgment. This pipeline—game transcript to features to probability—is what carries the claim.

What would settle it

Collect direct amusement ratings from human annotators on a few thousand Wordle games and retrain the same logistic regression on those labels; if accuracy is not significantly above 50%, or if the features that matter change substantially, the claim that game features predict user amusement would be unsupported.

Watch

Extended reading notes

Core claim

On its own terms, the core discovery is that amusement at Wordle games is predictable to a very modest extent from the games: a logistic regression with only aggregate game-level features classifies GPT-labeled amused versus not-amused reactions correctly 54.5% of the time on a balanced 15,000-game test set, where guessing the majority class would give 50%. The features that matter are game length, the reduction in the number of words still possible after guesses, edit distance between consecutive guesses, and how funny the guessed words are on their own. In particular, shorter games, larger final reductions in the candidate set, and a larger distance between the last two guesses predict more amusement, while average edit distance predicts less, and GloVe semantic distance has no measurable effect. The authors acknowledge the amusement labels are likely very imperfect, since GPT-3.5's agreement with human annotators is weak, and hypothesize that better labels would improve predictive performance.

Load-bearing premise

The load-bearing premise is that GPT-3.5's binary amused/not-amused judgments on Reddit comments are a trustworthy stand-in for what human players actually find amusing, even though agreement between GPT-3.5 and the human annotators is weak (Cohen's kappa about 0.16–0.40).

Editorial extensions

If this is right

Shorter games and large late reductions in the number of possible answers both predict amusement, suggesting that visibly lucky or skilled solves are a main emotional trigger.
A one-standard-deviation increase in last-guess Levenshtein distance raises predicted amused probability by about 4%, and a one-standard-deviation increase in the final answer-space reduction raises it by about 2%.
Game length alone is within 0.2 percentage points of the full model's accuracy, so the number of guesses is a near-sufficient summary of the signal in this feature set.
Average Levenshtein distance predicts less amusement while last-guess distance predicts more, so the final dramatic jump to the answer matters more than overall guess style.
The authors hypothesize that higher-quality amusement labels from humans would make the game features more predictive than they appear here.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial extension: because the labels are GPT-3.5's, the 54.5% figure may partly measure how predictable GPT-3.5's humor judgments are; a direct human-labeled replication is needed to confirm the signal attaches to human amusement.
Editorial extension: if the signal is real, puzzle generators could optimize for amusement by selecting Wordle answers that produce short games with dramatic collapses of the candidate set, turning humor into a tunable game-design objective.
Editorial extension: the small gap over baseline suggests most of the amusement signal lives in the comment text and social context rather than the game transcript, so combining game features with text features is a natural next step.
Editorial extension: the same feature family could be tested on other constrained word games (Quordle, Semantle, or custom Wordle variants) to see whether the pattern generalizes beyond one game's answer list.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear [Colleague],

Here's the short version: this paper shows that a handful of hand-crafted game features (number of guesses, reduction in possible answers, word-funniness norms, etc.) can predict whether GPT-3.5 calls a Reddit comment about a Wordle game 'amusing,' with 54.5% accuracy on a balanced test set of 15,000 games, against a 50% baseline. The effect is statistically significant and the authors are appropriately modest in the conclusions. The problem is the abstract's last sentence goes a step further: 'user amusement is predictable.' That isn't what they measured.

What's new: nobody has tried to predict amusement at Wordle games from game statistics before. The dataset scraping and the feature engineering are straightforward but sensible, and the statistical reporting is unusually careful for a short paper—p-values, standard errors, a Bonferroni mention, and a balanced test set. The funniness regression using Westbury & Hollis features is also a reasonable transfer.

The soft spots. The target label is GPT-3.5's binary 'amusing/not amusing' judgment. The agreement between GPT-3.5 and the five human annotators is weak (Cohen's Kappa 0.16–0.40, Table 1). The authors acknowledge this in the Conclusions, but the abstract and the final claim don't carry the same caveat. The stress-test note about the prompt is mostly wrong: the system prompt actually defines humor as 'the extent that the commenter is amused by the Wordle game.' So the construct is not as mismatched as the note claims. However, the weak Kappa still means we don't know how well GPT's labels track human amusement, and every result in the paper is about those labels.

Second, the full model is only 0.2% better than a univariate model using just the number of guesses. That tells you the signal is dominated by game length, not by the clever features. The authors report this honestly, but it undercuts the 'creativity infused through humor' framing.

Third, no code or data is released, and the few-shot examples are referenced but not shown in the text (they're in an appendix that isn't included here). That makes it hard to reproduce or evaluate the prompt quality.

Who is this for? People working on computational humor or LLM-based annotation. It's a useful data point, but not a smoking gun. It deserves a serious referee—the methodology is sound enough, and the label-validity issue is exactly what peer review should hammer on. I'd recommend sending it to review, with the expectation that the authors need to either validate against human labels or rewrite the claims to be about GPT-3.5's judgments.

Referee Report

4 major / 6 minor

Summary. The paper explores whether objective properties of a Wordle game (number of guesses, reductions in the space of possible answers, Levenshtein and GloVe distances between consecutive guesses, and a regression-based measure of intrinsic word funniness) can predict whether a Reddit comment about that game expresses amusement. The authors scrape about 80,000 game-comment pairs from r/Wordle, obtain binary amusement labels from GPT-3.5 via few-shot prompting, validate those labels against five human annotators (reporting weak to moderate Cohen's Kappa), and train logistic regression and neural network classifiers on balanced subsampled data. The best model achieves 54.5% correct classification on a balanced test set of 15,000 examples, with a univariate model using only game length within 0.2% of the full model. The paper concludes that user amusement in Wordle is computationally predictable to a modest extent.

Significance. If the central claim were supported, the paper would provide a modest but interesting empirical result: objective game statistics carry a weak signal about perceived humor, complementing prior work on perceived move brilliance in chess. The authors are transparent about their limitations, report coefficient tables with standard errors and p-values, and compare against a univariate baseline, which are strengths. The feature set is grounded in psycholinguistic humor norms and previous Wordle-behavior studies. However, the current experiments only establish that GPT-3.5's amusement labels are predictable; the abstract's claim about 'user amusement' is not yet supported because the agreement between GPT-3.5 and human annotators is weak (Kappa 0.158–0.398), and no evaluation of the final model on human labels is provided. The negligible difference from the game-length-only baseline further tempers the substantive significance of the feature-level findings.

major comments (4)

[Abstract and Conclusions] The central claim that 'user amusement at Wordle games can be predicted computationally' is not established by the reported experiments, because the prediction target is GPT-3.5's binary label, not human amusement. Table 1 reports Cohen's Kappa between GPT-3.5 and each of the five human annotators in the range 0.158–0.398 (weak agreement), and the Conclusions describe the rating as 'likely very imperfect.' The 54.5% accuracy and the coefficients in Table 2 therefore characterize the model's ability to predict GPT-3.5's judgments; they do not, by themselves, show that human amusement is predictable. I recommend either evaluating the fitted logistic regression on held-out human amusement labels (e.g., the author-annotated comments, supplemented if necessary), or explicitly limiting the paper's claims to prediction of GPT-annotated amusement.
[Model and Performance / Results] The full model's improvement over a single-feature baseline is negligible in practical terms: a univariate logistic regression using only num_possible_guesses_length achieves a correct classification rate that is only 0.2% lower than the full model. With a test set of 15,000 balanced examples, a chi-squared test can detect such a small difference as statistically significant, but the incremental predictive value of the remaining features is essentially zero. Since the paper's discussion interprets multiple features (Levenshtein distance, GloVe distance, intrinsic word humor) as contributing to amusement, the authors should report the univariate model's accuracy explicitly and provide effect sizes (e.g., accuracy difference, AUC) for the full versus univariate model. They should also consider whether the game-length signal is an artifact of comments that simply report the score rather than expressing amusement.
[Appendix (few-shot prompt)] The operational definition used to obtain labels is inconsistent with the paper's construct. The system prompt defines humor as 'the extent that the commenter is amused by the Wordle game,' but the instruction to GPT-3.5 asks whether 'the comment is funny' (0 or 1). These are different judgments: a comment can be funny without the commenter being amused by the game (e.g., a witty complaint), and a commenter can be amused while writing a non-funny comment. Unless the authors provide evidence that these two phrasings are empirically interchangeable, the reported labels are ambiguous with respect to the construct of interest. The prompt should be aligned with the intended construct, or the paper should acknowledge and justify the conflation.
[Dataset] The dataset construction introduces avoidable label noise: the authors state that the scraped replies 'often, though not always, are reactions to the Wordle game in the original post.' If a substantial fraction of replies are not responses to the game posted in the original post, their amusement cannot be predicted from game features, which dilutes the measured signal. The paper should report the proportion of replies that are direct top-level reactions to the game post, and either restrict the analysis to those or analyze the sensitivity of the results to this filtering.

minor comments (6)

[Abstract] The phrase 'verify that GPT-3.5's labels roughly correspond to human labels' overstates the agreement reported in Table 1 (Kappa 0.158–0.398); suggest 'weakly correspond' or similar.
[Feature: Intrinsically Funny Words] The reported 'R2 of 0.37022' should be formatted as 'R^2 = 0.370' and the decimal precision is excessive; also, the RMSE of 7.67 over a range of 72.3 is a useful effect-size indicator and should be stated as such.
[Results] The claim that 'the significant p-values are generally very small, and would be robust to a Bonferroni correction' is not accurate for all significant entries in Table 2: the coefficient for 'num possible guesses reduction mean' has p = 0.0396, which exceeds the Bonferroni threshold of approximately 0.00385 for 13 predictors, and 'levenshtein distance max' (p = 0.0577) is not significant at the 0.05 level. Please specify which p-values survive the correction.
[Model and Performance] The sentence 'Performance on the test set, for all the settings described above, was 54%±0.5%, using all the settings above' is ambiguous: it is unclear whether this is the best, average, or representative performance across settings. Please report the selection procedure and the variability across the three architectures and regularization options.
[Reference list] The reference 'CMLOEGCMLUIN. 2012. Relative frequencies of English phonemes' appears to contain a garbled author name; please verify the source.
[Feature: Luck or Skill] The interpretive claim that a larger Levenshtein distance 'might imply the user is having fun with different guesses' is speculative and not supported by the data; consider softening or removing such motivational language from the feature descriptions.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: game features are computed independently of the GPT-3.5 amusement labels; the weak human agreement is a validity concern, not a circular derivation.

full rationale

The derivation chain is: scrape Reddit comments; label them via few-shot GPT-3.5; compute game features (number of guesses, reductions in possible guesses, Levenshtein/GloVe distances, and an externally trained word-funniness score); fit a logistic regression to predict the labels; report 54.5% accuracy on a balanced held-out set. None of these steps defines a feature in terms of the label or vice versa. The intrinsic-humor feature comes from a linear regression trained on the Engelthaler and Hills (2018) humor norms, not on the Wordle amusement labels. The self-citations (Zaidi and Guerzhoy 2024; Liang et al. 2024) are used only as motivation and are not the basis of the predictive result. The paper's own Conclusion states the rating is 'likely very imperfect,' and Table 1 reports Cohen's kappa 0.158–0.398 between GPT-3.5 and the authors; this is a construct-validity limitation that weakens the inference to human amusement, but it is not circularity because the fitted coefficients come from external features and a held-out label source. No equation or definition in the paper makes a predicted quantity equivalent to an input by construction, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim relies on two main assumptions: the validity of GPT-3.5 as an annotator, which is only weakly supported, and the relevance of the hand-crafted features, which the data suggest is largely captured by game length. No new entities are introduced.

free parameters (3)

Few-shot examples in GPT-3.5 prompt = not specified
The prompt code references 'examples_text' but the actual examples are not shown; the chosen examples affect the labels.
Threshold for human amusement label = rating > 2 on a 1-5 scale
The authors thresholded human ratings to get binary labels; this choice affects the Kappa values and the ground truth.
L2 regularization strength for funniness regression = not specified
The funniness model uses L2 regularization; the strength is not reported, affecting the word humor scores.

assumptions (3)

domain assumption GPT-3.5's amusement labels correspond to human amusement.
The paper verifies weak agreement (Kappa 0.16-0.40) in Table 1, but the assumption is load-bearing for the central claim and is acknowledged as 'very imperfect'.
domain assumption The computed game features (reduction in possible guesses, Levenshtein and GloVe distances, word funniness) capture the relevant aspects of skill, luck, and humor.
The features are hypotheses, and the paper's own data show that a single feature (number of guesses) carries most of the signal, so the richer feature set is not strongly justified.
domain assumption The computation of the number of logically possible guesses at each game state is correct.
This computation is not described in detail, but the drop-in-possible-guesses features depend on it.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Automatically Detecting Amusing Games in Wordle." pith.science (2026). https://pith.science/paper/AGXE4S6E

@misc{pith2026250605415,
  author       = {Pith},
  title        = {Pith review of: Automatically Detecting Amusing Games in Wordle},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AGXE4S6E}},
  note         = {Machine review of arXiv:2506.05415}
}

read the original abstract

We explore automatically predicting which Wordle games Reddit users find amusing. We scrape approximately 80k reactions by Reddit users to Wordle games from Reddit, classify the reactions as expressing amusement or not using OpenAI's GPT-3.5 using few-shot prompting, and verify that GPT-3.5's labels roughly correspond to human labels. We then extract features from Wordle games that can predict user amusement. We demonstrate that the features indeed provide a (weak) signal that predicts user amusement as predicted by GPT-3.5. Our results indicate that user amusement at Wordle games can be predicted computationally to some extent. We explore which features of the game contribute to user amusement. We find that user amusement is predictable, indicating a measurable aspect of creativity infused into Wordle games through humor.

Figures

Figures reproduced from arXiv: 2506.05415 by the authors.

**Figure 2.** Actual and predicted humor ratings of the 4858 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

12 extracted references · 11 canonical work pages

[1]

Carnegie Mellon University . 1993. The Carnegie Mellon Pronouncing Dictionary . http://www.speech.cs.cmu.edu/cgi-bin/cmudict. Version 0.7b

work page 1993
[2]

CMLOEGCMLUIN. 2012. Relative frequencies of E nglish phonemes

work page 2012
[3]

Engelthaler, T., and Hills, T. T. 2018. Humor norms for 4,997 english words. Behavior research methods 50:1116--1124

work page 2018
[4]

Hollis, G.; Westbury, C.; and Lefsrud, L. 2017. Extrapolating human judgments from skip-gram vector representations of word meaning. Quarterly Journal of Experimental Psychology 70(8):1603--1619

work page 2017
[5]

Levenshtein, V. 1966. Binary codes capable of correcting deletions, insertions, and reversals. Proceedings of the Soviet physics doklady

work page 1966
[6]

Liang, G.; Kabbara, A.; Liu, C.; Luo, R.; Kim, K.; and Guerzhoy, M. 2024. Semantic, orthographic, and morphological biases in humans' wordle gameplay. arXiv preprint arXiv:2411.18634

work page arXiv 2024
[7]

Pennington, J.; Socher, R.; and Manning, C. D. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP) , 1532--1543

work page 2014
[8]

practicalcryptography.com. 2024. English letter frequencies

work page 2024

Show all 12 references

[9]

Tatman, R. 2017. English word frequency

2017
[10]

Westbury, C., and Hollis, G. 2019. Wriggly, squiffy, lummox, and boobs: What makes some words funny? Journal of Experimental Psychology: General 148(1):97

2019
[11]

Zaidi, K., and Guerzhoy, M. 2024. Predicting user perception of move brilliance in chess. ICCC 2024

2024
[12]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Carnegie Mellon University . 1993. The Carnegie Mellon Pronouncing Dictionary . http://www.speech.cs.cmu.edu/cgi-bin/cmudict. Version 0.7b

work page 1993

[2] [2]

CMLOEGCMLUIN. 2012. Relative frequencies of E nglish phonemes

work page 2012

[3] [3]

Engelthaler, T., and Hills, T. T. 2018. Humor norms for 4,997 english words. Behavior research methods 50:1116--1124

work page 2018

[4] [4]

Hollis, G.; Westbury, C.; and Lefsrud, L. 2017. Extrapolating human judgments from skip-gram vector representations of word meaning. Quarterly Journal of Experimental Psychology 70(8):1603--1619

work page 2017

[5] [5]

Levenshtein, V. 1966. Binary codes capable of correcting deletions, insertions, and reversals. Proceedings of the Soviet physics doklady

work page 1966

[6] [6]

Liang, G.; Kabbara, A.; Liu, C.; Luo, R.; Kim, K.; and Guerzhoy, M. 2024. Semantic, orthographic, and morphological biases in humans' wordle gameplay. arXiv preprint arXiv:2411.18634

work page arXiv 2024

[7] [7]

Pennington, J.; Socher, R.; and Manning, C. D. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP) , 1532--1543

work page 2014

[8] [8]

practicalcryptography.com. 2024. English letter frequencies

work page 2024