{"id":"763b8597-6b15-43de-af62-2c28a6d62a5a","arxiv_id":"2506.00814","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A Minecraft-based benchmark shows vision-language models often fail to decode player-built creations, with accuracy falling sharply for rare concepts and low-resource languages.","lead":"GuessBench is a new benchmark that gives vision-language models 2,000 guessing problems built from 1,500 screenshots of the Minecraft minigame 'Guess the Build'. It finds that even the strongest tested model, GPT-4o, misses about one third of the dynamic problems, and that accuracy drops sharply for open models, rare concepts, and low-resource languages.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Construct validity is the load-bearing risk: synthetic letter-mask hints and curation may make GuessBench a word-puzzle-plus-recognition test rather than a creativity benchmark.","rationale":"The paper has real strengths: it collects actual gameplay screenshots, reports a transparent evaluation setup, and its difficulty numbers are internally consistent with the reported model configurations. However, the paper's contribution is not merely a new VQA dataset; it is a benchmark for 'creativity in the wild.' That construct depends on two design decisions in §2.2: the use of synthetic letter-mask hints instead of real player hints, and the removal of exactly the noisy, non-representational builds that define 'in the wild' behavior. The text provides no validation that the synthetic hints are informationally equivalent to the real hint exchange, no statistics on the curation filter, and no human baseline. The most direct counter-explanation is that GPT-4o's 57.8% reflects difficulty in object recognition plus a masked-word puzzle, not difficulty in creative sensemaking. This is more load-bearing than the 15.36% transfer claim: if the construct validity fails, the benchmark's interpretation and the transfer-based 'training resource' claim lose their foundation, even though the raw numbers are correct. My proposed test—replacing the synthetic hints with actual human hints and re-measuring GPT-4o accuracy—would settle this directly. I therefore agree with the reader's weakest assumption and see no reason to change the CONDITIONAL verdict.","tokens_in":27953,"tokens_out":5605,"duration_ms":56913,"concrete_test":"Collect or reconstruct actual in-game hints for the 500 build sets (e.g., by re-recording rounds on Hypixel for the same concepts, or by having human annotators provide free-form hints to a guesser for each build image). Re-run the static GPT-4o evaluation exactly as in Table 2, but replacing the synthetic letter-mask hint with the real hint. If static accuracy changes by more than ~10 percentage points, or if the image-only-vs-full-task gap changes materially, then the synthetic hint mechanism is load-bearing and GuessBench is not a faithful test of in-the-wild sensemaking.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that GuessBench measures 'sensemaking creativity in the wild' rests on two unvalidated choices in §2.2. First, the hints are synthetic: the paper does not use the actual natural-language hints from the game, but generates letter masks from the answer word (lengths, then ⌈N/8⌉ random letters, then ⌈N/4⌉). A player's real hint—e.g., 'it's a bug, it's green'—has a very different information structure from a masked spelling puzzle: the letter mask gives exact positional constraints that turn the problem into a word search, while real hints are noisy, ambiguous, and socially grounded. The paper provides no evidence that these synthetic masks preserve the difficulty or the cognitive operation of human hinting. Second, the curation removes builds where players spell the answer in blocks, or where 'no meaningful construction is present'—precisely the kind of low-effort, noisy, and idiosyncratic player behavior that 'in the wild' creativity is supposed to include. No statistics are given on how many builds were removed or whether the remaining 500 are representative of the game's creative space. Since there is also no human baseline, GPT-4o's 57.8% could reflect the difficulty of the masked-word puzzle plus Minecraft-block recognition rather than the difficulty of decoding another person's creative intent. If either of these assumptions fails, the benchmark's headline conclusions—challenging creativity task, open-vs-API gap, transfer gains—are still numerically correct but no longer support the 'creativity' interpretation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"GuessBench collects 500 Minecraft build sets (1500 screenshots) from the Hypixel minigame \"Guess the Build\" and defines static and dynamic VLM guessing tasks with progressively revealed hints that are synthetic letter masks generated from the answer word. The paper evaluates six VLMs, reports that GPT-4o achieves 57.8% static and 66.0% dynamic accuracy while open models lag far behind, and claims that fine-tuning on GuessBench reasoning traces improves visual perception tasks by 15.36% on average. Additional analyses examine multilingual robustness, concept frequency, multi-view inputs, sycophancy, in-context learning, and generation-versus-discrimination performance. The authors position the benchmark as a testbed for sensemaking creativity in the wild.","tokens_in":28191,"tokens_out":6033,"duration_ms":55088,"significance":"If the construct-validity concerns about the synthetic hints, curation, and scoring policy are resolved, GuessBench would be a valuable resource: the images come from real multiplayer gameplay with anonymization, the static/dynamic design allows measuring iterative refinement, the single-modality ablations in Table 2 cleanly support the claim that multimodal integration is required, and the experimental details in Appendix C are sufficiently concrete to reproduce. The paper also makes several useful empirical observations (concept-frequency correlation, language correlation, sycophancy, in-context learning) that stand independently of the \"creativity\" framing. Its current significance, however, is capped by the lack of evidence that the task measures creative sensemaking rather than masked-word solving plus Minecraft object recognition; the headline difficulty and transfer numbers cannot be fully interpreted without a human baseline and a scoring policy for multiple valid answers.","major_comments":[{"comment":"The hints used in GuessBench are synthetic letter masks generated from the answer word (word lengths, then ⌈N/8⌉ random letters, then ⌈N/4⌉ random letters), not the natural-language hints that real players exchange in \"Guess the Build\". The abstract and Figure 1 describe \"natural language hints of varying completeness\", but the actual prompts are templated statements such as \"The 5th letter is r\". A real hint such as \"it's a green bug\" has a fundamentally different information structure from a positional letter constraint, and the letter mask gives exact positional evidence that turns the problem into a word search. Since the central claim is that GuessBench measures sensemaking creativity in the wild, the paper needs either to collect or replicate real player hints, or to justify with human experiments that the synthetic masks preserve the difficulty and cognitive operation of human hinting; otherwise the headline difficulty numbers reflect a word-puzzle-plus-recognition task rather than decoding another person's creative intent.","section":"§2.2, Hint Generation"},{"comment":"The curation explicitly removes builds where players spell out the answer with blocks or where \"no meaningful construction is present\". These are precisely the noisy, low-effort, and idiosyncratic player behaviors that \"creativity in the wild\" is supposed to include, yet no statistics are given for how many builds were removed or how the remaining 500 sets represent the game's creative space. The paper also provides no human baseline on the curated 500 sets, so the interpretation of GPT-4o's 57.8% static accuracy as \"uniquely challenging\" is underdetermined: without human accuracy, the reader cannot tell whether the benchmark is hard because it measures creative decoding or because the synthetic hints and curation make the task artificially constrained.","section":"§2.2, Build Collection"},{"comment":"The paper acknowledges in §4 and Appendix H.2 that some GuessBench questions have multiple valid answers, but the evaluation protocol appears to be exact-match after GPT-4o answer extraction (Appendix C.2) and no policy is stated for counting alternative valid answers as correct. If a model outputs a different but valid construction (e.g., \"Finger of God\" for a build that is also \"Modern Art\"), it is presumably scored incorrect. This directly affects all reported accuracy numbers in Table 2 and the open-versus-API gap, and it is not an edge case: the paper's own case study shows a correct initial answer being abandoned during image retrieval precisely because of this ambiguity. The authors should specify the scoring rule, provide a set of accepted answers per problem (or measure inter-annotator agreement on answer validity), and report accuracy under that rule.","section":"§2.3 and §4 (evaluation of multiple valid answers)"},{"comment":"The abstract and introduction claim that fine-tuning on GuessBench reasoning traces \"improves visual perception tasks by 15.36% on average\", but §5.5 reports a decline on MathVista and gains only on MultiChartQA and BLINK, and the 15.36% figure does not appear in §5.5 or Figure 6. The average presumably covers only the two positive benchmarks; as stated, the claim is misleading because a reader would expect an average over all evaluated target tasks. Please report per-benchmark deltas, the exact set of tasks included in the average, variance or repeated seeds, and clarify whether MathVista's decline is included. With 289 training samples, one base model, and one seed, the transfer result also needs additional conditions to support the general claim.","section":"§5.5, Figure 6"}],"minor_comments":[{"comment":"The phrase \"start-of-the-art\" should be \"state-of-the-art\", and Table 2 contains the typo \"Alldenotes\" for \"All denotes\".","section":"Abstract and Introduction"},{"comment":"The evaluation section defines accuracy but does not state how near-synonyms or paraphrases (e.g., \"graveyard\" versus \"cemetery\") are handled; this should be specified alongside the exact-match rule.","section":"§2.3"},{"comment":"The Figure 2 caption says \"higher is better\" for all three AutoBencher metrics, but Difficulty is defined as the lowest error rate; the direction of each metric should be made explicit to avoid confusion.","section":"§5.1, Figure 2"},{"comment":"In the provided manuscript, the text inside Figures 2, 3, 4, and 5 is rendered as Unicode escape sequences, making the plots unreadable; please ensure the figures are legible in the submission.","section":"Figures 2–5"},{"comment":"Using GPT-4o as the answer extractor for responses generated by GPT-4o introduces a possible scoring bias; report extraction accuracy on a human-annotated sample to show that the extraction step is not systematically favoring or penalizing particular models.","section":"Appendix C.2"},{"comment":"The manual evaluation that yields 80% generative accuracy reports no inter-annotator agreement or detailed rubric for \"plausibly constructed within the Minecraft environment\", so the comparison to the 40% discriminative accuracy should be interpreted cautiously.","section":"Appendix E"},{"comment":"The paper does not state where the 1500 images and annotations will be released; a public data link or a clear availability statement is needed for the benchmark to be usable by the community.","section":"Data Availability"}],"recommendation":"major_revision","confidential_remarks":"The paper's framing as \"creativity in the wild\" is currently stronger than the evidence. If the authors add real-player hints, a human baseline, and a multiple-answer scoring policy, GuessBench would be a solid contribution; otherwise the contribution is better described as a challenging multimodal word-puzzle benchmark on Minecraft builds. The abstract's transfer-learning claim should be corrected to avoid cherry-picking two positive benchmarks, given that MathVista declines in the same experiment. I also recommend asking the authors to double-check that all figure text is embedded as vector text rather than font-dependent glyphs, since several figures were illegible in the submitted version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"GuessBench is a legitimate new benchmark idea, not a routine rehash. The dataset construction and evaluations are transparent, and the core observation—VLMs, including GPT-4o, struggle on these guessing problems and open models struggle much more—holds up. But the paper currently overclaims what it measures. The hints are not real player hints but synthetic letter masks derived from the answer word. That changes the task from decoding noisy, socially grounded creative intent to solving a masked-word puzzle on top of Minecraft-object recognition. Without real hints or a validation that synthetic masks preserve the information structure of human hints, the 'sensemaking creativity in the wild' label is not fully earned. The curation also removes 'low-quality' builds with no counts, and the builds removed are exactly the noisy cases the 'in the wild' framing depends on. There is no human baseline, so 57.8% is hard to interpret.\n\nWhat the paper does well: the static/dynamic design is a real task contribution; the single-modality ablations support the multimodal-integration conclusion; the language-robustness and concept-frequency analyses are useful and independently checkable; prompts, decoding parameters, and fine-tuning details are reported in full. The internal difficulty numbers are consistent with Table 2.\n\nSoft spots beyond construct validity: the 15.36% transfer claim in the abstract is oversold. Section 5.5 shows gains on BLINK and MultiChartQA but a decline on MathVista; averaging the first two and calling it 'visual perception tasks' needs the caveat. No dataset or code is released; for a benchmark paper that is a serious limitation, not just logistics. Using GPT-4o both as evaluated model and answer extractor is a mild self-reference, not fatal, but worth a sanity check with another extractor. The limitations section acknowledges geographic skew and the static-image simulation but is silent on the synthetic hints and curation dropout, which is where the biggest threats sit.\n\nBottom line: this deserves a serious referee, but the verdict should be conditional. I would accept for review and ask for data/code release, a human baseline or real hints, counts and representativeness of the curation, and an honest re-framing of the transfer result. In current form it is a useful benchmark for object-plus-letter-puzzle reasoning; with those fixes it could genuinely be a creativity benchmark.","headline":"GuessBench is a genuine new benchmark idea with mostly transparent execution, but the synthetic letter-mask hints and missing human baseline make the 'creativity sensemaking' label provisional, and the 15.36% transfer number is oversold.","tokens_in":28789,"tokens_out":3389,"would_cite":false,"duration_ms":34734,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces GuessBench, a 2,000-problem benchmark built from real Minecraft 'Guess the Build' games, and claims that even the strongest tested vision-language model fails on about a third of attempts to say what a player built.","keywords":["GuessBench","vision-language models","creativity evaluation","Minecraft","multimodal benchmark","sensemaking","concept frequency","multilingual robustness"],"falsifier":"Have human players from the same 'Guess the Build' community solve the same 500 static problems and the same 500 three-attempt dynamic problems with the same synthetic hints; if human accuracy is no better than the best model's 57.8% and 66.0%, the difficulty claim could be an artifact of underdetermined puzzles rather than a VLM creativity deficit.","tokens_in":27713,"feed_emoji":"🧩","tokens_out":8984,"duration_ms":86783,"temperature":0.7,"pith_summary":"The paper sets out to prove that 'creativity in the wild'—the imperfect, everyday creativity of ordinary people rather than trained artists—can be measured by asking vision-language models to guess what a player's Minecraft build is meant to represent. It curates 500 real in-game builds into 2,000 static and dynamic problems, where both the image and the hint become progressively more complete. The central empirical claim is that current models largely fail this task: the strongest tested model reaches 57.8% accuracy in the static setting and 66.0% on the final dynamic attempt, while open models average only about 14% on that attempt. The paper argues these failures reveal a real capability gap, and supports that reading by showing that success tracks concept frequency in training data, drops for low-resource languages and non-Western cultural symbols, and is not fixed by standard reasoning strategies. It further claims the benchmark is useful for training, since fine-tuning on its reasoning traces improves downstream visual-perception tasks by 15.36% on average.","feed_headline":"Best AI misses a third of Minecraft creative-build guesses","feed_subtitle":"A benchmark drawn from live 'Guess the Build' games shows models struggle to decode human creativity, not just recognize objects.","key_machinery":"The load-bearing object is the GuessBench problem set: 500 curated build sets, each providing three progressively complete screenshots of the same Minecraft construction and a synthetic hint that encodes word lengths and letter positions. On the first dynamic attempt the hint shows only the answer's word/letter structure; later attempts reveal $\\lceil N/8 \\rceil$ and then $\\lceil N/4 \\rceil$ randomly selected letters for an $N$-letter answer, with the static setting using the final image and hint from this progression. This design turns an open-ended interpretive act into a repeatable measurement: models must integrate visual evidence with partial text across attempts, and correctness is scored by extracting the guessed concept from the model's free-form response. The same response traces serve as fine-tuning data, which is the mechanism behind the reported transfer to visual-perception tasks.","core_discovery":"GuessBench's central claim is that decoding another person's creative construction—figuring out what a block-based build represents from its appearance and a sparse hint—is a distinct, testable skill that vision-language models do not yet have. In the static setting the model sees one finished build plus a hint revealing the answer's word and letter structure; the best model tested is correct 57.8% of the time. In the dynamic setting it sees three progressively complete builds and increasingly revealing hints; on the third attempt the best model reaches 66.0%, meaning 34% of instances remain wrong. The paper also reports a stark open-versus-API gap, transfer gains from fine-tuning on the benchmark's reasoning traces, strong correlations between accuracy and concept frequency in pretraining data, and a tendency for models to abandon correct answers when a user falsely insists they are wrong.","pith_inferences":["The paper does not include a human baseline; recruiting players from the same community to solve the identical problems would show how much of the 34% error floor is task underdetermination rather than model limitation.","Because the hints are synthetic letter patterns rather than the semantic, often playful hints real players give, part of what GuessBench measures may be constraint satisfaction over spelling; a version using recorded human hints would separate these abilities.","If the transfer result is robust, the dynamic setting could serve as a cheap automatic curriculum: models could be trained on progressively informative image-hint pairs, with difficulty controlled by how many letters are revealed.","The cultural-bias finding suggests a concrete fairness test: supplementing training data with builds and answers from low-resource languages and non-Western contexts should shrink the reported accuracy gap if the benchmark is sensitive to representation rather than to some artifact of translation."],"forward_implications":["If GuessBench measures what it claims, current vision-language models are much weaker at decoding creative intent than their strong scores on conventional visual question-answering benchmarks suggest.","The large gap between open and API models implies that strong performance on earlier benchmarks may come partly from memorization of similar data rather than from general visual interpretation.","Because accuracy tracks concept frequency and language resources, improving creative sensemaking will require targeted data or retrieval support for long-tail concepts, non-Western symbols, and low-resource languages, not just larger models.","Fine-tuning on GuessBench reasoning traces should transfer to perception-heavy tasks, while tasks requiring precise logical or mathematical reasoning may degrade, matching the paper's reported MathVista decline.","The contradiction experiments imply that interactive uses—where users push back on a model's answer—are risky in creative settings unless the model can check whether its own guess is right."],"supporting_citations":[{"why":"Supplies the answer-extraction protocol used to score model guesses and serves as a comparison baseline in the difficulty analysis.","marker":"[48]"},{"why":"Defines the Difficulty, Separability, and Novelty metrics used to position GuessBench against existing benchmarks.","marker":"[41]"},{"why":"Provides corpus frequency statistics from a 2.6-trillion-token corpus for the concept-frequency correlation analysis.","marker":"[44]"},{"why":"Is one of the downstream visual-perception benchmarks used in the transfer fine-tuning comparison.","marker":"[18]"},{"why":"Is the other downstream visual-perception benchmark used to measure transfer gains from GuessBench fine-tuning.","marker":"[88]"},{"why":"Is the self-refinement reasoning approach tested as an enhancement strategy; its limited gain supports the claim that existing methods do not close the gap.","marker":"[52]"},{"why":"Is the image-retrieval intervention whose failures illustrate the many-to-one mapping problem in creative sensemaking.","marker":"[87]"}],"fun_headline_variants":["GPT-4o fails a third of Minecraft build-guess challenges","New benchmark from Minecraft 'Guess the Build' shows AI creativity gap","AI vision models struggle to decode creative builds in live game","GuessBench: best AI only 66% accurate on Minecraft creativity test","Minecraft data reveals AI fails on creative inference, not just vision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's load-bearing premise is that its image-plus-letter-hint problems measure creative sensemaking rather than object recognition combined with a spelling puzzle.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4o fails a third of Minecraft build-guess challenges","New benchmark from Minecraft 'Guess the Build' shows AI creativity gap","AI vision models struggle to decode creative builds in live game","GuessBench: best AI only 66% accurate on Minecraft creativity test","Minecraft data reveals AI fails on creative inference, not just vision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000361,"raw_usage":{"total_tokens":1959,"prompt_tokens":963,"completion_tokens":996,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":906}},"tokens_in":579,"tokens_out":996,"duration_ms":9506,"temperature":1.0,"reasoning_tokens":906,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:57:26.400461+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human players from the same 'Guess the Build' community solve the same 500 static problems and the same 500 three-attempt dynamic problems with the same synthetic hints; if human accuracy is no better than the best model's 57.8% and 66.0%, the difficulty claim could be an artifact of underdetermined puzzles rather than a VLM creativity deficit.","supporting_citations":[{"cited_title":"Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts","cited_arxiv_id":null,"evidence_quote":"Supplies the answer-extraction protocol used to score model guesses and serves as a comparison baseline in the difficulty analysis."},{"cited_title":"Autobencher: Towards declarative benchmark construction","cited_arxiv_id":null,"evidence_quote":"Defines the Difficulty, Separability, and Novelty metrics used to position GuessBench against existing benchmarks."},{"cited_title":"Infini- gram: Scaling unbounded n-gram language models to a trillion tokens","cited_arxiv_id":null,"evidence_quote":"Provides corpus frequency statistics from a 2.6-trillion-token corpus for the concept-frequency correlation analysis."},{"cited_title":"Guess the Build","cited_arxiv_id":null,"evidence_quote":"Is the other downstream visual-perception benchmark used to measure transfer gains from GuessBench fine-tuning."},{"cited_title":"Self-refine: Iterative refinement with self-feedback.Advances in Neural Information Processing Systems, 36:46534–46594, 2023","cited_arxiv_id":null,"evidence_quote":"Is the self-refinement reasoning approach tested as an enhancement strategy; its limited gain supports the claim that existing methods do not close the gap."},{"cited_title":"Enhancing interactive image retrieval with query rewriting using large language models and vision language models","cited_arxiv_id":null,"evidence_quote":"Is the image-retrieval intervention whose failures illustrate the many-to-one mapping problem in creative sensemaking."}],"review_version":1}