Pith. sign in

REVIEW 3 major objections 6 minor 27 references

AGI Is Coming... Right After AI Learns to Play Wordle

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A computer-using AI agent solves Wordle only 5.36% of the time, and its reported color recognition collapses from 42% on the first guess to 6% by the fifth.

desk verdict CUA's 5.36% Wordle success rate is a solid empirical data point; the color-perception story is weakened because the accuracy figures come from the model's own tool-call notes, not a controlled perception probe. read the letter →

arxiv 2504.15434 v1 pith:U4CEVBLE submitted 2025-04-21 cs.AI cs.CV

Sarath Shekkizhar , Romain Cosentino This is my paper · ORCID
classification cs.AIcs.CV
keywords computer-usingagentmultimodalagentsWordlecolorperceptionGUIinteractionbenchmarkevaluationoptimisticbiasvisualtokenization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a state-of-the-art computer-using agent can play Wordle, a deceptively simple five-letter word game in which every guess is answered with colored tiles. It reports that the agent solves the puzzle only 5.36% of the time over 200 games, and that the main failure is contextual color perception: the color feedback the agent records for its own guesses is correct 42% of the time on the first attempt and 6% of the time on the fifth. Gray tiles are frequently described as yellow or green, so the agent often announces a win that the game never granted. The authors take this as evidence that strong performance on complex computer-use benchmarks does not imply mastery of simple, multi-step perceptual-reasoning tasks, and that this gap matters for claims about approaching general intelligence.

What carries the argument

The load-bearing mechanism is the agent's self-annotation tool, `update_wordle_game_state`, which forces the model to convert what it sees in each screenshot into a five-character string of green, yellow, and black feedback after every guess. The paper scores these observation strings against the true feedback the game should have produced, turning an invisible perceptual process into a measurable trace. A secondary mechanism is the screenshot tokenizer: each $1024 \times 768$ image is split into a $2 \times 2$ grid of $512 \times 512$ patches, with the inferred boundary cutting through letter position 3 and attempt 3, the same regions where the agent's reported color accuracy is lowest. This tokenization hypothesis, plus the contrast between accurate isolated color tests and inaccurate in-game color reports, carries the paper's argument that the failure is contextual rather than a simple inability to see colors.

What would settle it

Run the agent on 100 pre-filled Wordle screenshots that contain no live game state, ask it to call the same annotation tool on each grid, and compare its reported green/yellow/black strings to the true tile colors: if accuracy stays near ceiling across grids that mimic attempts 1 through 5, the decay reported here is tied to acting and chain-of-thought rather than to seeing colors; if accuracy still falls on later-looking grids, the perceptual-failure reading is confirmed.

Watch

Extended reading notes

Core claim

The central finding is a context-dependent color-recognition failure in a production computer-using agent. Over 200 runs across eight days of Wordle puzzles, the agent solved the game in 5.36% of cases, averaging 3.25 guesses on the rare successful solves. When asked to record the color feedback it saw after each guess, the agent's reported accuracy fell from 42% at the first attempt to 6% at the fifth, with edge letter positions more reliable than central ones. The dominant error pattern was gray tiles reported as yellow or green, which the paper reads as an optimistic bias that lets the agent believe it has finished. Word-level success rates tracked color-observation accuracy ($r = 0.694$, $p = 0.056$), making the perceptual errors a direct driver of game failure.

Load-bearing premise

The entire color-accuracy analysis assumes the agent's self-reported observation strings are a faithful record of what it perceives, rather than an optimistic story it tells about its own guesses; the authors explicitly note this alternative in the paper.

Editorial extensions

If this is right

  • High scores on complex web and computer-use benchmarks do not predict success on a simple GUI task like Wordle.
  • The agent's color-observation accuracy decays across attempts, so perceptual reliability should be measured as a function of context length and accumulated screen content, not just overall task success.
  • Because gray-to-green hallucinations make the agent declare victory early, agent evaluation should audit the agent's internal state reports, not only whether the final answer is correct.
  • The position-dependent accuracy pattern predicts that changing how screenshots are cropped or tokenized, for example by zooming into the grid, would change Wordle performance in a measurable way.
  • Word-level variation with a correlation of $r = 0.694$ between color accuracy and success suggests that the same agent can appear competent or incompetent depending on which puzzle it draws.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A clean extension would freeze a completed Wordle grid in front of the agent and ask for annotations with no gameplay in between; if accuracy stays high, the failure belongs to acting under accumulated context, while a continued decay would point to the image tokenizer itself.
  • The paper's accuracy numbers inherit whatever bias the agent has in its self-reports, so an independent pixel-level comparison of the screenshots against the agent's recorded feedback would separate 'cannot see the color' from 'reports an optimistic color'.
  • AGI-style capability claims are usually tested on end-task success; this work suggests a cheaper audit: probe the perceptual state an agent reports before each action, because that trace can expose failures that final outcomes hide.
  • The same measurement pattern could be applied to other grid-based games or interfaces with deterministic visual feedback, such as Minesweeper, Sudoku, or form validators, to see whether context-dependent color or icon misreading generalizes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper reports an empirical evaluation of OpenAI's Computer-Using Agent (CUA) on the New York Times Wordle game. Over 200 runs across 8 days, the agent solved the puzzle in only 5.36% of cases. The authors analyzed the color feedback the agent reported through the update_wordle_game_state tool and found that self-reported color-recognition accuracy declines from 42% on attempt 1 to 6% on attempt 5, with gray tiles most often misreported as yellow or green and with higher accuracy at edge positions. They report a correlation between per-word color-observation accuracy and success rate (r=0.694, p=0.056), interpret these patterns as context-dependent perceptual failure, and discuss tokenization, RLHF bias, and implications for agentic AI evaluation.

Significance. If the perceptual claim were supported, this would be a valuable, falsifiable counterexample to benchmark-based optimism about agentic AI, backed by a clean 200-run autonomous evaluation, explicit prompts and tool definitions, and control tests for simple color vision. The paper's strengths are the reproducible protocol skeleton, the honest acknowledgment of the alternative bias explanation in Section 3.5, and the large number of runs behind the success-rate statistic. However, the central perceptual-failure conclusion currently rests on a self-report proxy, so the significance is conditional on adding an independent perception probe or reframing the claim as a state-tracking failure.

major comments (3)
  1. [§3.3–3.5, Figs. 2–4] All color-accuracy figures are computed from the observation field that CUA writes into update_wordle_game_state, which the Appendix A system prompt instructs the model to use to 'record your thinking and carefully analyze the screenshot.' This is a free-text, post-decision summary, not a calibrated perceptual readout, and the authors themselves note in §3.5 that 'the model might just be biased toward seeing specific colors even when gray.' Because the decline from 42% to 6% and the common Gray→Yellow/Gray→Green errors are measured on this self-report proxy, the headline claim of a context-dependent perceptual failure is not established; the evidence as presented supports a claim about the agent's self-reported state tracking, which could be distorted by guess-rationalization or optimistic bias. Please either add a screenshot-only perception probe that queries tile colors before a guess is committed, or reframe the central claim accordingly.
  2. [§3.1, Alg. 1, Appendix A] The manuscript does not state how the expected feedback (the ground-truth G/Y/B string for each guess) was obtained, nor how screenshots were aligned to tile states; the exclusion of the sixth attempt 'because at times the screenshot was not timed appropriately' (fn. 2) shows that this alignment is nontrivial. Without a documented ground-truth procedure, the accuracy numbers in Figs. 2–4 and Table 1 are not reproducible. Please provide the source of the target word, the parsing or alignment rule used to derive expected colors, and the timing criterion used to drop the sixth attempt.
  3. [§3.6, Fig. 5] With only eight word-level data points, r=0.694 and p=0.056, the evidence for the correlation between color-observation accuracy and success rate is suggestive but not significant at the conventional 5% level, and no confidence interval is reported. Calling this 'a strong validation on the hypothesis' overstates the result; please report bootstrapped confidence intervals for the correlation and temper the language to match the statistical power.
minor comments (6)
  1. [§3.4–3.5] The text refers to 'Fig. 4 (Left)' in §3.4 and §3.5 for content that appears in Fig. 3; please correct the figure cross-references throughout.
  2. [Appendix A] The prompt and tool definition contain formatting artifacts such as 'Ob je ct iv e', 's u b s e q u e n t', 'minLe ng th', and 'm ax Le ng th'; please clean these up.
  3. [Table 1] Table 1 gives the mean guesses per solved puzzle but not the number of solved puzzles (about 11 of 200) or a measure of spread; please report the sample size and standard deviation or range.
  4. [Abstract vs. §3.1] The abstract says 'several hundred runs' while §3.1 reports 200 runs; please use the exact number or describe the additional runs explicitly.
  5. [Figs. 2–4] The accuracy plots would benefit from error bars or confidence intervals; with self-report data, point estimates alone overstate precision.
  6. [Fig. 2 (right)] The tokenizer patch boundary in Fig. 2 (right) is admittedly 'deduced from ... documentation'; please label it as a hypothesis in the figure caption as well as in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the Wordle evaluation is an empirical measurement with no fitted parameters, no self-citation chain, and no target result assumed as input.

full rationale

This paper is an empirical evaluation, not a derivation with equations or fitted models. The central finding—declining color-observation accuracy across attempts—is computed by comparing the CUA agent's self-reported observation strings (via the update_wordle_game_state tool) against the Wordle ground-truth feedback. This is a direct behavioral measurement of what the agent reports seeing, not a parameter fitted to a subset of data and then renamed as a prediction. No success metric or conclusion is assumed as an input to the analysis. The paper explicitly acknowledges the main interpretative alternative: 'Alternatively, the model might just be biased toward seeing specific colors even when gray' (Section 3.5), which turns the self-report proxy concern into a stated limitation rather than a hidden circular step. There are no load-bearing self-citations, no imported uniqueness theorems, and no renaming of a known result under new coordinates. The correlation between observation accuracy and success rate (r = 0.694) is a descriptive statistic over the collected runs, not a fitted prediction. Even the skeptical concern about self-reports is a validity threat to the perceptual claim, not a circularity in the paper's derivation chain. Therefore no enumerated circularity pattern applies, and the paper is self-contained as an empirical study with score 0.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper's central empirical claims rest on five assumptions: the game's feedback is the ground truth, the agent's self-reported observations reflect its perception, the model is out-of-distribution on Wordle, the vision encoder's patch layout is as inferred, and the evaluation protocol is fair. None of these are proven; the first two are the most load-bearing for the color-accuracy analysis.

assumptions (5)
  • domain assumption The Wordle game's color feedback is treated as the ground truth for the agent's observations.
    Sections 3.2-3.5 compare the agent's reported observations to the expected colors; this assumes the game's feedback is correct and that expected colors can be computed from the daily answer.
  • domain assumption The agent's observation strings from the update_wordle_game_state tool accurately reflect its visual perception.
    All color accuracy metrics are based on comparing the model's tool call observations to ground truth; the model might be biased by its own guesses, as acknowledged in Section 3.5.
  • domain assumption CUA was not trained on Wordle gameplay.
    Section 2.2 assumes the NYT terms of service prevent bot scraping, making Wordle an out-of-distribution task; this underlies the generalization discussion but is not verifiable.
  • ad hoc to paper The vision encoder splits 1024x768 screenshots into four 512x512 patches arranged in a 2x2 grid.
    Section 3.3 and Fig. 2 infer patch boundaries from OpenAI documentation and model output metadata; the authors flag it as deduction, and the position-dependent accuracy pattern is interpreted through this lens.
  • domain assumption The evaluation protocol (prompt, tool definitions, API settings) does not artificially degrade the agent's performance.
    Section 3 states design choices were 'thoroughly validated and evaluated for optimality' but provides no comparative data; a different prompt could yield different success rates.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AGI Is Coming... Right After AI Learns to Play Wordle." pith.science (2026). https://pith.science/paper/U4CEVBLE

@misc{pith2026250415434,
  author       = {Pith},
  title        = {Pith review of: AGI Is Coming... Right After AI Learns to Play Wordle},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U4CEVBLE}},
  note         = {Machine review of arXiv:2504.15434}
}
abstract

This paper investigates multimodal agents, in particular, OpenAI's Computer-User Agent (CUA), trained to control and complete tasks through a standard computer interface, similar to humans. We evaluated the agent's performance on the New York Times Wordle game to elicit model behaviors and identify shortcomings. Our findings revealed a significant discrepancy in the model's ability to recognize colors correctly depending on the context. The model had a $5.36\%$ success rate over several hundred runs across a week of Wordle. Despite the immense enthusiasm surrounding AI agents and their potential to usher in Artificial General Intelligence (AGI), our findings reinforce the fact that even simple tasks present substantial challenges for today's frontier AI models. We conclude with a discussion of the potential underlying causes, implications for future development, and research directions to improve these AI systems.

Figures

Figures reproduced from arXiv: 2504.15434 by the authors.

Figure 1
Figure 1. Visualization of Wordle game played by the CUA agent – During a game session, the CUA agent is instructed to self-annotate or summarize the screen (via function calling). At the displayed state of the game, the CUA agent describes the screen as follows: “The grid shows attempts with first guess (S T A R E) displaying two letters in yellow (S and T), 2nd guess (L E E T S) shows green for E, 3rd guess (S I Z E S) reta… view at source ↗
Figure 2
Figure 2. (Left) CUA color observation accuracy by letter position and attempt - Heatmap showing color observation (as identified by the CUA model while playing the game) accuracy by letter position (1-5) and attempt number (1-5). Data shows highest accuracy (94%) at position 1, attempt 1, with significant degradation in accuracy in later attempts and central positions. (Right) Depiction of the potential image tokenizer patch… view at source ↗
Figure 3
Figure 3. (Left) Model-generated color observation accuracy by attempt number. The average accuracy (bold black line) decreases dramatically from 42% in attempt 1 to 6% in attempt 5, with individual words showing varying decline patterns. This demonstrates the agent’s increasing difficulty in correctly perceiving colors as the game progresses. Note that the accuracy in the first attempt, although relatively higher than the re… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: (Left) Success rate by word - Bar chart showing success rate by target Wordle word. ARROW (13.6%) and TURBO (10.0%) had the highest success rates, while SHEAR and WHEAT had a 0% success rate. The pattern closely mirrors observation accuracy, supporting the connection b…
Figure 5
Figure 5. Figure 5: Correlation between observation color accuracy and word success rate - Scatter plot showing the correlation between observation accuracy and success rate across different words. With a Pearson r of 0.694 and p-value of 0.056, there is a strong positive correlation betw…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 11 canonical work pages

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Agashe, S., Han, J., Gan, S., Yang, J., Li, A., and Wang, X. E. (2024). Agent s: An open agentic framework that uses computers like a human. arXiv preprint arXiv:2410.08164

    arXiv 2024

  3. [3]

    Introducing computer use

    Anthropic (2024). Introducing computer use. Anthropic Blog. https://www.anthropic.com/news/3-5-models-and-computer-use

    work page 2024

  4. [4]

    Benveniste, A. (2022). The sudden rise of wordle. The New York Times . https://www.nytimes.com/2022/01/31/crosswords/nyt-wordle-purchase.html

    work page 2022

  5. [5]

    a is b" fail to learn

    Berglund, L., Tong, M., Kaufmann, M., Balesni, M., Stickland, A. C., Korbak, T., and Evans, O. (2023). The reversal curse: Llms trained on" a is b" fail to learn" b is a". arXiv preprint arXiv:2309.12288

    arXiv 2023

  6. [6]

    Chen, X., Wu, Z., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., and Ruan, C. (2025). Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811

    arXiv 2025

  7. [7]

    Chowdhury, N., Johnson, D., Huang, V., Steinhardt, J., and Schwettmann, S. (2025). Investigating truthfulness issues in a pre-release o3 model. https://transluce.org/investigating-o3-truthfulness

    work page 2025

  8. [8]

    Gambardella, A., Iwasawa, Y., and Matsuo, Y. (2024). Language models do hard arithmetic tasks easily and hardly do easy arithmetic tasks. arXiv preprint arXiv:2406.02356

    arXiv 2024

Show all 27 references
  1. [9]

    He, H., Yao, W., Ma, K., Yu, W., Dai, Y., Zhang, H., Lan, Z., and Yu, D. (2024). Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919

    2024 arXiv

  2. [10]

    Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al. (2022). Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221

    2022 arXiv

  3. [11]

    Leng, J., Huang, C., Zhu, B., and Huang, J. (2024). Taming overconfidence in llms: Reward calibration in rlhf. arXiv preprint arXiv:2410.09724

    2024 arXiv

  4. [12]

    Liu, H., Li, C., Wu, Q., and Lee, Y. J. (2023). Visual instruction tuning. Advances in neural information processing systems , 36:34892--34916

    2023

  5. [13]

    Llama 4 models

    MetaAI (2025). Llama 4 models. Llama Blog. https://www.llama.com/models/llama-4/

    2025

  6. [14]

    Pixtral large

    MistralAI (2024). Pixtral large. Mistral Blog. https://mistral.ai/news/pixtral-large

    2024

  7. [15]

    Browsecomp: a benchmark for browsing agents

    OpenAI (2025a). Browsecomp: a benchmark for browsing agents. OpenAI Blog. https://openai.com/index/browsecomp/

    2025

  8. [16]

    Computer-using agent

    OpenAI (2025b). Computer-using agent. OpenAI Blog. https://openai.com/index/computer-using-agent/

    2025

  9. [17]

    Introducing operator: Our first ai agent that can use computers

    OpenAI (2025c). Introducing operator: Our first ai agent that can use computers. OpenAI Blog. https://openai.com/blog/introducing-operator

    2025

  10. [18]

    Petrov, I., Dekoninck, J., Baltadzhiev, L., Drencheva, M., Minchev, K., Balunovi \'c , M., Jovanovi \'c , N., and Vechev, M. (2025). Proof or bluff? evaluating llms on 2025 usa math olympiad. arXiv preprint arXiv:2503.21934

    2025 arXiv

  11. [19]

    Qin, Y., Ye, Y., Fang, J., Wang, H., Liang, S., Tian, S., Zhang, J., Li, J., Li, Y., Huang, S., et al. (2025). Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326

    2025 arXiv

  12. [20]

    R., Cheng, N., Durmus, E., Hatfield-Dodds, Z., Johnston, S

    Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., Cheng, N., Durmus, E., Hatfield-Dodds, Z., Johnston, S. R., et al. (2023). Towards understanding sycophancy in language models. arXiv preprint arXiv:2310.13548

    2023 arXiv

  13. [21]

    and Sun, H

    Wang, B. and Sun, H. (2025). Is the reversal curse a binding problem? uncovering limitations of transformers from a basic generalization failure. arXiv preprint arXiv:2504.01928

    2025

  14. [22]

    Wang, G., Ge, Y., Ding, X., Kankanhalli, M., and Shan, Y. (2023). What makes for good visual tokenizers for large language models? arXiv preprint arXiv:2305.12223

    2023 arXiv

  15. [23]

    Wolfe, J. M. (2020). Visual search: How do we find what we are looking for? Annual review of vision science , 6(1):539--562

    2020

  16. [24]

    and Xie, S

    Wu, P. and Xie, S. (2023). V*: Guided Visual Search as a core mechanism in multimodal LLMs. In CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2023

  17. [25]

    J., Cheng, Z., Shin, D., Lei, F., et al

    Xie, T., Zhang, D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., et al. (2024). Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems , 37:52040--52094

    2024

  18. [26]

    Xu, Y., Wang, Z., Wang, J., Lu, D., Xie, T., Saha, A., Sahoo, D., Yu, T., and Xiong, C. (2024). Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint arXiv:2412.04454

    2024 arXiv

  19. [27]

    F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., et al

    Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., et al. (2023). Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854

    2023 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.