Pith. sign in

REVIEW 4 major objections 6 minor 25 references

GuessArena: Guess Who I Am? A Self-Adaptive Framework for Evaluating LLMs in Domain-Specific Knowledge and Reasoning

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GuessArena claims that a 30-card yes/no guessing game, with cards auto-extracted from domain documents, ranks LLMs on domain knowledge and separates knowledge gaps from reasoning gaps.

desk verdict A useful evaluation framework whose central knowledge-vs-reasoning claim is undercut by a leaked deck-of-cards prompt; still worth refereeing. read the letter →

arxiv 2505.22661 v1 pith:FBZOOOK6 submitted 2025-05-28 cs.CL

Qingchen Yu , Zifan Zheng , Ding Chen , Simin Niu , Bo Tang , Feiyu Xiong , Zhiyu Li This is my paper
classification cs.CL
keywords LLMevaluationdomain-specificknowledgeinteractivereasoningbenchmarkadaptabilityapplicabilityefficiencyyes/noquestiongames
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that static LLM benchmarks cannot adapt to new domains and cannot distinguish missing knowledge from weak reasoning. To fix this, it proposes GuessArena, which turns user-supplied domain documents into a deck of 30 keyword cards per domain and has each tested model identify a hidden card through yes/no questions answered by a judge model. The framework scores models on how many cards they guess correctly, how few questions they need relative to a random baseline, and how well they use domain knowledge during the game. Across finance, healthcare, manufacturing, information technology, and education, the paper reports that the composite score ranks nine LLMs and that chain-of-thought versus knowledge-injected prompts reveal which weakness drives low performance. If the claim holds, domain-specific evaluation can be assembled from documents rather than hand-built benchmarks.

What carries the argument

The load-bearing mechanism is the "Guess Who I Am?" protocol adapted to domain evaluation. A deck of 30 cards is sampled from clusters of keywords extracted from domain documents, a judge model answers each query with only Yes, No, Invalid, or End, and the tested model alternates questioning and guessing until it names the hidden card. The composite score is a weighted average of reasoning accuracy (the share of correct final guesses), reasoning efficiency (a sigmoid penalty on question count relative to a random baseline), and knowledge applicability (an exponential penalty for steps beyond that baseline). This mechanism is what lets the framework measure information-gathering strategy rather than only final answer correctness.

What would settle it

Replay the full evaluation with a deliberately weak judge model or with human judges answering the same dialogue histories and compare the resulting rankings; or delete clearly invalid cards such as the OCR artifacts "uxsinodb" and "nvdia vgpu" from the IT deck and check whether the model ordering changes.

Watch

Extended reading notes

Core claim

The central claim is that an adversarial yes/no guessing game, with cards extracted automatically from domain documents, produces a composite score that distinguishes LLMs on domain knowledge coverage and on the completeness of their reasoning chains. The paper treats the game as a two-stage evaluation: domain knowledge modeling builds the deck, and interactive reasoning evaluation records each model's questions, guesses, and steps. The reported results show that the strongest evaluated models keep high scores across all three prompting regimes, while weaker models improve selectively under chain-of-thought or knowledge-driven prompts, which the paper takes as evidence that the two failure modes can be separated.

Load-bearing premise

The framework assumes the automatically constructed 30-card deck is a valid, discriminative test for the domain, that the judge model's yes/no answers are reliable ground truth, and that the random baseline used for efficiency scoring is well-defined.

Editorial extensions

If this is right

  • A new domain can be evaluated by feeding in documents, with no manual question annotation or answer key.
  • Decks can be regenerated from fresh documents, reducing the risk of benchmark leakage that static tests face.
  • Comparing basic, chain-of-thought, and knowledge-driven prompts reveals whether a model's weak domain performance comes from reasoning limits or knowledge gaps, guiding targeted improvements.
  • Because the judge model is an LLM, the same pipeline can be run for any testee model, making evaluation cheap enough to repeat as models are updated.
  • The published framework and decks allow others to rerun the same five-domain evaluation or build their own.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: a control condition with generic or irrelevant injected knowledge would test whether score gains under the knowledge-driven prompt come from filling real knowledge gaps or simply from additional context.
  • Editorial inference: deck quality is likely a major noise source, since cards such as "uxsinodb" and "nvdia vgpu" in the IT deck are not clean domain terms; filtering or entity-linking them could change rankings even if the framework's logic is sound.
  • Editorial inference: because judge reliability was checked on only 1,200 sampled instances, long-tail domains may need an ensemble of judges or a user-supplied oracle to keep the score trustworthy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes GuessArena, a self-adaptive evaluation framework in which an LLM must identify a hidden card by asking yes/no questions about a deck extracted from domain documents. The framework has two stages: automatic construction of domain card decks from unstructured documents using GPT-4o keyword generation, embedding filtering, and spectral clustering; and multi-turn interactive evaluation where a judge model answers the tested model's questions. The authors define a composite score combining reasoning accuracy, reasoning efficiency, and knowledge applicability, and report results for nine LLMs across five domains under basic, chain-of-thought, and knowledge-driven prompting. They claim that the framework effectively distinguishes LLMs in domain knowledge coverage and reasoning chain completeness, and that the three prompt conditions can separate knowledge deficits from reasoning deficits. The paper also reports a human-annotation consistency study for the judge model and a small case study.

Significance. If the central claims were supported, GuessArena would address a real and timely problem: building adaptive, domain-specific LLM evaluations that are less vulnerable to static-benchmark leakage than conventional datasets. The game-based protocol offers interpretable traces of questioning behavior, and the open-sourced implementation is a concrete practical contribution. The human-annotation validation of judge answers on 1,200 instances is a genuine strength, and the paper is honest about some limitations in its final section. However, the load-bearing evidence for separating knowledge from reasoning is confounded by the construction of the knowledge-driven prompt, and the reasoning metrics are not fully specified. These issues currently prevent the paper from establishing its headline claims.

major comments (4)
  1. [§4.2, Appendix C.2, Fig. 10] The knowledge-driven prompt is generated from the exact 30-card evaluation deck: the template in Figure 10 instructs GPT-4o to generate a background 'based on the following cards: {deck_of_cards}' and to 'Provide logical clues that can be used to differentiate between cards.' A tested model can therefore improve in Table 4 simply by matching clue phrases to card labels rather than by supplying missing domain knowledge. Because the contrast between the CoT condition and the knowledge-driven condition is the paper's main evidence for dissociating 'domain knowledge coverage' from 'reasoning chain completeness,' this confound undermines the central claim. The knowledge background should be generated from domain source documents that are independent of the test deck, or the experiment should be rerun with a held-out knowledge source.
  2. [§3.3, Eqs. (6)-(7)] The metric t_rand, described as 'the number of reasoning steps required by a random baseline,' is never defined anywhere in the paper. Eq. (6) also references a hyperparameter α that does not appear in the formula, and Eq. (7) defines K purely as a function of t_model/t_rand, making K and F numerically redundant rather than a measure of knowledge applicability. The authors need to define the random baseline concretely (e.g., expected elimination time under uniform random branching), specify how α is set or remove it, and justify why K measures knowledge rather than an efficiency transformation.
  3. [§4.1, Tables 2-4] Each domain uses only 30 fixed cards, and no error bars, repeated sampling, or significance tests are reported. Scores differing by roughly 0.01-0.02 (e.g., GPT-4o 0.9244 vs. Qwen2.5-72B 0.9052 in Info Tech, Table 2) are interpreted as meaningful distinctions, but with a single 30-card run the variance is unknown. The claim that GuessArena 'effectively distinguishes' LLMs requires either multiple independent card samples with variance reporting or appropriate statistical tests.
  4. [§3.1, §4.3, Appendix C.1] GPT-4o is simultaneously the judge, an evaluated model, the generator of card keywords, and the generator of the knowledge background. The human-annotation agreement study in §4.3 helps validate judge answers, but it does not validate the quality of the automatically constructed decks or the neutrality of the knowledge generation. The IT deck in Appendix C.1 contains clear artifacts such as 'uxsinodb' and 'nvdia vgpu', and 'business challenges' appears in the finance deck; without human screening of card quality or an independent judge for the final rankings, the framework's validity as a domain-knowledge measure remains questionable.
minor comments (6)
  1. [Table 1] There is a typo in the caption: 'listsists' should be 'lists', and 'NAN' should be 'N/A'.
  2. [Eq. (3)] The filtering thresholds τl and τu are said to be 'empirically determined via grid search', but no grid-search details, validation data, or sensitivity analysis are provided.
  3. [§3.3, Eq. (5)] The definition of E as 'correct guesses / total guesses' is ambiguous: if 'total guesses' refers to the number of final guesses rather than the number of rounds, a model that guesses early could be penalized. The sentence should clarify that each round contributes exactly one final guess.
  4. [Fig. 3] The figure is readable but the small font and dense grouping make cross-model comparison difficult; a table with deltas and an explicit statement of which differences are statistically meaningful would be more informative.
  5. [§4.3, Table 5] The majority-voting row is labeled 'Majority V oting' with an extra space; this should be corrected.
  6. [Limitations] The limitation paragraph mentions judge bias but does not acknowledge that the knowledge background is constructed from the same deck used for evaluation; this confound should be explicitly discussed.

Circularity Check

2 steps flagged · score 6.0 of 10

Knowledge-driven prompt is generated from the exact 30-card deck and instructed to provide differentiating clues, so its gains do not establish a knowledge gap; the K metric also renames step count as knowledge applicability.

  1. self definitional [Appendix C.2 (Figure 10) and Section 4.2]
    ""Your task is to generate a concise and informative domain knowledge background based on the following cards: {deck_of_cards}" and "Provide logical clues that can be used to differentiate between cards" (Figure 10). Section 4.2: "for models lacking corresponding domain knowledge, the provision of relevant background information significantly improves performance"."

    The 'knowledge' injected in the knowledge-driven condition is generated from the exact 30-card evaluation deck, with an explicit instruction to supply logical clues that differentiate those cards. It is therefore not independent domain knowledge but a clue sheet keyed to the test set. The improvement under this prompt, interpreted in Section 4.2 as evidence of a domain-knowledge gap, is in part an artifact of revealing the deck's distinguishing structure; models can improve by matching background terms to card labels. The claimed dissociation between knowledge coverage and reasoning chain completeness therefore reduces, by construction, to the way the prompt input was created.

  2. renaming known result [Section 3.3, Eqs. (6)-(7)]
    ""Reasoning efficiency (F) ... F = 1 / (1 + exp(4 · (t_model − t_rand) / t_rand))" and "Knowledge applicability (K) ... K = exp(−max(0, (t_model − t_rand) / t_rand))"."

    F and K are both monotone functions of the same ratio (t_model − t_rand)/t_rand. K therefore measures step efficiency, not knowledge applicability; it merely renames the variable already used in F with a different transform. The composite score (Eq. 4), which sums E, F, and K, does not integrate an independent knowledge-utilization signal, so the claim that the framework separates knowledge coverage from reasoning efficiency is not established by the metric definitions.

full rationale

The core circularity is in the knowledge-driven experiment: the background knowledge is constructed from the exact test deck with instructions to provide differentiating clues, so gains under it cannot be cleanly attributed to filling a knowledge gap. The judge-model validation against human annotations (Table 5) provides independent grounding for the oracle answers, so the GPT-4o-as-judge design is not itself circular. The basic-prompt ranking may still carry information, but the paper's central claim of separating knowledge coverage from reasoning chain completeness rests on the confounded knowledge-driven comparison and on a K metric that is a deterministic transform of the same step-count ratio used in F. These are partial, construction-level reductions rather than a fully forced equivalence, hence a score of 6.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The framework rests on several hand-chosen parameters (embedding thresholds, cluster count, card count, scoring weights, and an efficiency penalty constant) and on assumptions about the quality of automatically extracted cards and the reliability of GPT-4o as judge. No new physical or ontological entities are introduced.

free parameters (6)
  • cosine similarity thresholds tau_l, tau_u = 0.35, 0.9
    Eq. (3): thresholds 'empirically determined via grid search' to filter keywords.
  • number of clusters = 10
    Section 3.1: spectral clustering groups keywords into 10 categories; chosen by authors.
  • number of cards per domain = 30
    Section 4.1: 30 cards extracted from each knowledge base; no justification given.
  • efficiency penalty constant = 4
    Eq. (6): F = 1/(1+exp(4*(t_model-t_rand)/t_rand)); text mentions alpha but the formula hard-codes 4.
  • random baseline t_rand = unspecified
    Eqs. (6)-(7) rely on t_rand, the number of steps for a random baseline, but the paper never defines how it is computed.
  • composite weights w1, w2, w3 = 1/3 each
    Section 3.3: set to equal contribution; no sensitivity analysis provided.
assumptions (5)
  • domain assumption The collected Fortune Global 500 documents are representative of each target domain
    Appendix B.1: 128 documents from annual reports and news; representativeness is assumed.
  • domain assumption GPT-4o's keyword extraction and judge responses are reliable and unbiased
    Sections 3.1-3.2: GPT-4o generates keywords and answers Yes/No; only a one-time 1,200-instance consistency check is provided.
  • domain assumption The random baseline t_rand is a meaningful reference for reasoning efficiency
    Eqs. (6)-(7): efficiency and knowledge metrics are defined relative to t_rand, which is never specified.
  • domain assumption Automatically extracted cards are valid, discriminative test items
    Appendix C.1: card lists contain OCR artifacts (uxsinodb, nvdia vgpu) and generic phrases (business challenges), so validity is questionable.
  • domain assumption The LLM-generated knowledge background does not leak the target card
    Section 4.2 and Figure 10: knowledge-driven prompts supply GPT-4o-generated background; leakage or clueing is not tested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GuessArena: Guess Who I Am? A Self-Adaptive Framework for Evaluating LLMs in Domain-Specific Knowledge and Reasoning." pith.science (2026). https://pith.science/paper/FBZOOOK6

@misc{pith2026250522661,
  author       = {Pith},
  title        = {Pith review of: GuessArena: Guess Who I Am? A Self-Adaptive Framework for Evaluating LLMs in Domain-Specific Knowledge and Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FBZOOOK6}},
  note         = {Machine review of arXiv:2505.22661}
}
read the original abstract

The evaluation of large language models (LLMs) has traditionally relied on static benchmarks, a paradigm that poses two major limitations: (1) predefined test sets lack adaptability to diverse application domains, and (2) standardized evaluation protocols often fail to capture fine-grained assessments of domain-specific knowledge and contextual reasoning abilities. To overcome these challenges, we propose GuessArena, an adaptive evaluation framework grounded in adversarial game-based interactions. Inspired by the interactive structure of the Guess Who I Am? game, our framework seamlessly integrates dynamic domain knowledge modeling with progressive reasoning assessment to improve evaluation fidelity. Empirical studies across five vertical domains-finance, healthcare, manufacturing, information technology, and education-demonstrate that GuessArena effectively distinguishes LLMs in terms of domain knowledge coverage and reasoning chain completeness. Compared to conventional benchmarks, our method provides substantial advantages in interpretability, scalability, and scenario adaptability.

Figures

Figures reproduced from arXiv: 2505.22661 by the authors.

Figure 1
Figure 1. Illustration of the "Guess Who I Am?" game. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Framework of GUESSARENA. The framework comprises two core components: Domain Knowledge Modeling (Left Panel), which parses and models domain-specific documents to generate a candidate card repository for evaluation; and Interactive Reasoning Evaluation (Right Panel), which employs a multi-turn dialogue mechanism to construct an interactive reasoning game, systematically assessing the model’s key capability metrics. … view at source ↗
Figure 3
Figure 3. Cross-domain GUESSARENA scores (higher is better) for nine LLMs under three prompting strategies. Grouped bars show the composite GUESSARENA performance achieved with basic, cot, and knowledge-driven prompts in each of the five industry domains, allowing a visual comparison of prompt effectiveness across models and domains. ing the scores of nine LLMs across five domains under three prompting strategies. As illustra… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Interactive guessing trajectories in the healthcare scenario. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Prompt template for deriving domain-specific keywords that seed the G [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Prompt template issued to the judge model (oracle judge) in GUESSARENA [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Prompt template for the evaluated model (player) under the basic prompting regime in GUESSARENA. You will participate as a player in a card-guessing game. In front of you is a deck of cards: {deck_of_cards}, and your task is to guess the chosen card that has been secre…
Figure 8
Figure 8. Figure 8: Prompt template for the evaluated model (player) under the cot prompting regime in GUESSARENA [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Prompt template for the evaluated model (player) under the knowledge-driven prompting regime in GUESSARENA. You are an expert in the field of {name}, which primarily focuses on {description}. Your task is to generate a concise and informative domain knowledge backgroun…
Figure 10
Figure 10. Figure 10: Prompt template for generating the domain-level knowledge background used in the [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

25 extracted references · 23 canonical work pages

  1. [1]

    Domain Focus: Prioritize professional terms, concepts, or industry jargon from the field, ensuring the keywords accurately reflect the core content of the domain

    work page

  2. [2]

    product lifecycle management (plm)

    Uniqueness: Ensure that each keyword in the set is unique, avoiding synonyms or near-synonyms. Each keyword must be a complete term, avoiding any abbreviations or combinations with explanatory content. For example, avoid forms like “product lifecycle management (plm)”, “lms (learning management system)”, or similar formats. Keywords should be the full ter...

    work page

  3. [3]

    Broad Coverage: The keywords should broadly cover various knowledge aspects of the field, including common terms, basic concepts, and specialized vocabulary from subfields. If appropriate, you may also include well-known entities within the domain, such as company names, product names, and people’s names, as these entities are important representatives of...

    work page

  4. [4]

    capital markets

    Contextual Relevance: Ensure that there is an inherent connection between the keywords. For example, in the finance industry, "capital markets" might be related to "stocks," or "balance sheets" might be connected to "financial analysis." Make sure related keywords have sufficient contextual relevance

    work page

  5. [5]

    The final keywords should all be in English and separated by semicolons

    Diversity and Representativeness: Ensure that the keywords span different domains, levels, and knowledge points, covering not only basic concepts but also niche terms and specialized language unique to the industry. The final keywords should all be in English and separated by semicolons. Please do not include any addi- tional explanations, formatting, or ...

    work page

  6. [8]

    [Yes]",

    Response Rules: You can only use four standard responses: "[Yes]", "[No]", "[Invalid]", and "[End]". Do not provide any additional explanations. Maintain absolute objectivity in your responses and do not reveal any specific information about the chosen card

    work page

  7. [9]

    [Invalid]

    Question Determination: If a player’s question is outside the scope of the card deck or obviously unreasonable, respond with "[Invalid]"; if the question is clear and within the answerable range, respond with "[Yes]" or "[No]" based on the actual characteristics of the chosen card

    work page

  8. [10]

    Ensure that each question receives a fair and accurate judgment

    Consistency Principle: For substantially the same or highly similar questions, you must maintain consistent responses. Ensure that each question receives a fair and accurate judgment

    work page

Show all 25 references
  1. [11]

    [Yes]" or

    Game Termination: When the player gives the final guess of the chosen card, you need to immediately respond with "[End]". ## Judge’s Responsibilities - Maintain absolute neutrality, do not provide any additional hints - Strictly follow the above rules to ensure the fairness of...

  2. [15]

    [Invalid]

    *Invalid Behaviors: Repeating the same question or guess is prohibited. Any questions or guesses unrelated to the game will be marked as "[Invalid]" by the judge. Please begin by asking your first question. {dialog history} Figure 7: Prompt template for the evaluated model (pl...

  3. [19]

    [Invalid]

    Invalid Behaviors: Repeating the same question or guess is prohibited. Any questions or guesses unrelated to the game will be marked as "[Invalid]" by the judge. ## Strategy Suggestions

  4. [20]

    Step-by-Step Reasoning: Before each question, build a reasoning chain based on known information, clarify the current range of possible options, and choose the key question that best reduces uncertainty

  5. [21]

    Prioritize Key Features: Ask questions about features that can significantly differentiate most of the cards, quickly narrowing down the possible options

  6. [22]

    [Yes]" or

    Timely Guessing: Once the final chosen card is determined, avoid excessive questioning to prevent it from negatively impacting your score. Please begin by asking your first question. {dialog history} Figure 8: Prompt template for the evaluated model (player) under the cot prom...

  7. [23]

    [Yes]" or

    Question Rules: You may ask only one clear and concise question at a time, which must be answerable with "[Yes]" or "[No]". The question must not contain line breaks, nor can it directly ask about the card’s specific identity. Do not request additional hints

  8. [24]

    The fewer the questions, the higher the final score, assuming you correctly guess the chosen card

    Scoring Mechanism: The game score is inversely proportional to the number of questions asked. The fewer the questions, the higher the final score, assuming you correctly guess the chosen card. Successfully and quickly identifying the target card is key to achieving a high score

  9. [25]

    When you make your final guess for the chosen card, the judge will immediately respond with "[End]", regardless of whether the guess is correct

    Guessing Process: After each question, wait for the judge’s response, then use that information, along with previous questions, to ask the next question. When you make your final guess for the chosen card, the judge will immediately respond with "[End]", regardless of whether ...

  10. [26]

    [Invalid]

    Invalid Behaviors: Repeating the same question or guess is prohibited. Any questions or guesses unrelated to the game will be marked as "[Invalid]" by the judge. ## Knowledge Background The following is domain-specific knowledge related to these cards, which you can use as a r...

  11. [27]

    Relevance to Cards: Ensure the background content is directly related to the cards’ theme, describing the domain’s key characteristics, typical classifications, and common attributes

  12. [28]

    Logical Reasoning Support: Provide logical clues that can be used to differentiate between cards, but avoid giving overly specific or direct information

  13. [29]

    Ensure the knowledge background covers all the cards’ content without omitting any part

    Diversity and Representativeness: Highlight the domain’s diversity by mentioning different subfields, classifications, or representative concepts. Ensure the knowledge background covers all the cards’ content without omitting any part

  14. [30]

    The information provided should guide reasoning without favoring any specific card

    Neutrality and Accuracy: Maintain a neutral and objective tone. The information provided should guide reasoning without favoring any specific card

  15. [31]

    knowledge_background

    Clarity and Conciseness: The knowledge background should be concise (250-300 words) while ensuring clear language, rigorous logic, and accurate information that is easy to understand. Do not include any additional explanations or extraneous text. Output the domain knowledge ba...

  16. [2020]

    In International Conference on Learning Representations

    Measuring massive multitask language under- standing. In International Conference on Learning Representations. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Ja- cob Steinhardt. 2021. Measuring mathematical prob- lem solving ...

    2021 arXiv

  17. [2024]

    Guess Who I Am?

    Cs-bench: A comprehensive benchmark for large language models towards computer science mastery. arXiv preprint arXiv:2406.08587. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Se- bastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou,...

    2022 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.