Pith. sign in

REVIEW 3 major objections 5 minor 24 references

LivingArena: Do LLMs Know What Other LLMs Don't? Peer-Probing as Scalable Evaluation

T0 review · 3 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By letting frontier LLMs interrogate one another with judge-verified, freshly generated questions, LivingArena produces a stable Elo leaderboard—and shows that the models genuinely find and re-attack their peers' weak dimensions.

desk verdict A real and well-specified evaluation idea—peer models generating, validating, and grading questions in an Elo tournament—worth refereeing, but the objectivity claim rests on unvalidated judge consensus. read the letter →

arxiv 2607.24780 v1 pith:C736ZUBG submitted 2026-06-19 cs.AI

Xingyu Chen , Rui Wang , Zhaopeng Tu , Liefeng Bo This is my paper
classification cs.AI
keywords LLMevaluationpeerprobingadversarialtournamentEloratingbenchmarkcontaminationself-calibrationLLM-as-a-judgeknowledgeboundaries
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LivingArena is a proposal to replace static, human-written benchmarks with a zero-sum game in which frontier LLMs take turns asking each other questions that a panel of model judges must verify as objective and must grade as correct or wrong. The paper's central claim is that this peer-probing game yields a stable Elo leaderboard that separates models that static benchmarks can no longer distinguish, and that the separation is not an artifact: tournament logs show models localizing and re-attacking an opponent's weak dimension after a successful hit. If the claim holds, LLM evaluation can become self-adapting and contamination-resistant without human authoring, and it measures a distinct axis—objective rigor, factual calibration, and the higher-order ability to probe a peer's blind spots—that human preference voting does not capture. The framework also exposes practical diagnostics such as self-harm rate, hit rate as asker, and answer accuracy, which point to specific failure modes.

What carries the argument

The central object is LivingArena's bidirectional adversarial match: ten rounds of question-generation and answering in each direction (A asks B, then B asks A). The asker must submit a question, a gold answer, and verification logic; a panel of three contestant models vetoes any question that any judge finds invalid (self-harm penalty −1.0), the answerer is graded by majority CORRECT/WRONG, and successful hits are decay-weighted per dimension (1.0, 0.5, 0.2) to encourage breadth. Net scores map to a pairwise outcome, Elo is updated with K=32, and six capability dimensions give per-dimension diagnostics. The mechanism turns 'a question you cannot answer' into a verifiable fact/reasoning-trap

What would settle it

Sample, say, 200 questions and 200 graded responses from the tournament logs; have independent human experts—or executable checkers for code and math items—re-verify the gold answers and re-grade the responses, then compare with the judge panel's votes. If disagreement is large or correlates with the identity of the asker or answerer, the claimed objective-verification guarantee fails and the ranking cannot be trusted as a measure of ability.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is twofold. First, peer probing works as an evaluation: ten frontier models played a round-robin tournament of 360 matches (3,600 rounds), and the resulting Elo ratings separate five tiers with bootstrap-stable ranks, even where static benchmarks saturate. Second, the tournament logs show the mechanism is real: models answer their own validated questions in 93–100% of self-play rounds, yet re-target the same dimension after a hit 20.1% of the time versus 3.8% after a miss, and in unconstrained bonus rounds they attack a previously-hit weak dimension 41.4% versus 18.2% for never-hit dimensions, with 70.7% of bonus questions landing in discovered weak ar

Load-bearing premise

Around §3 and §5.4, the whole leaderboard rests on the assumption that three LLM contestants acting as judges can always decide whether a proposed question has a correct gold answer and whether a response matches it; if validation or grading errors are frequent or biased toward a model's family, Elo would rank judge preferences rather than objective ability.

Editorial extensions

If this is right

  • Because questions are generated fresh each round, the approach resists contamination from fixed question banks and can regenerate difficulty as models improve.
  • Asking and answering are separable axes; a two-axis capability map (hit rate vs. accuracy) reveals whether a model's deficit is knowledge, questioning strategy, or calibration.
  • Self-harm rate becomes a calibration diagnostic: a high rate identifies models that often produce incorrect or unverifiable gold answers, a failure mode that aggregate accuracy hides.
  • A full ten-model round-robin tournament runs in about 100 minutes with parallel API calls, making continuous, low-cost regression evaluation feasible without human authoring.
  • The per-dimension scoring and bonus-round logs can localize which capability dimension drove a model's rank, rather than giving only a single aggregate number.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: because the judges are themselves contestants, the framework's gate is the panel's reliability; an easy robustness check is to rerun the tournament with an independent, less capable judge set and see whether the Elo order changes.
  • Editorial extension: the paper's stated 'shared blind spot' limit means peer-probing cannot detect gaps common to all participants; injecting external verifiers (e.g., program execution, symbolic solvers, or retrieved sources) for selected dimensions would extend it beyond pure self-reference.
  • Editorial extension: the exploit-after-hit effect suggests a predictive use: before release, a model could be run against a small panel of peers, and the dimensions where they concentrate attacks could serve as a list of likely failure modes—testable against held-out human error data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes LivingArena, a fully automated evaluation framework in which frontier LLMs take turns posing questions to each other in a round-robin tournament. Each question must be accompanied by a gold answer and verification logic; a judge panel of three contestant models validates the question and later grades the answerer's response. The authors run 360 matches among ten models, compute an Elo leaderboard, and report behavioral analyses: models 'exploit-after-hit' (they revisit dimensions on which they previously stumped an opponent), and bonus rounds show strategic re-attack of discovered weaknesses. The paper claims the resulting leaderboard is contamination-resistant, self-adaptive, and measures an objective 'adversarial rigor' axis distinct from human preference.

Significance. If the judge-panel validity concern is resolved, LivingArena is a valuable contribution: it is fully automated, requires no static question bank, is scalable (~100 minutes for a ten-model tournament), and the authors release code and raw logs. The distinction between asking and answering ability and the self-harm calibration signal are genuinely useful diagnostics. The exploit-after-hit behavioral finding is interesting and, for stronger questioners, supported by effect sizes with bootstrap confidence intervals. The core risk is that the entire pipeline treats the three-model judge panel's consensus as objective ground truth, and this premise is never externally validated.

major comments (3)
  1. [§3 (Steps 2 & 4); §5.4; Limitations] The framework's objectivity premise rests on the judge panel (three LLM contestants) validating gold answers and grading responses. The paper reports high inter-judge agreement (Fleiss κ = 0.915 grading, 0.693 validation) and a self-bias audit, but agreement is not correctness. No sample of validated questions or graded responses is checked against human experts or an external benchmark. Concretely: if the three judges share a systematic misconception, a wrong gold answer is accepted, and an answerer giving the truly correct answer (contradicting the flawed gold) is graded WRONG. The Limitations section admits an 'evaluator ceiling' for shared blind spots, but applies it only to probing difficulty, not to gold-answer validity. I recommend a human-expert audit of a random sample (e.g., 200 questions across dimensions) to estimate the judge panel's error rate, and report panel-vs-external
  2. [§5.2 and §5.3 (Exploit-after-Hit, Sustained Probing)] The paper claims the exploit-after-hit effect is 'positive for all ten models' and that models 'double down on discovered vulnerabilities.' However, §5.3 states that for the Claude family and GPT-5.2 (weak questioners), the sample size of post-first-hit rounds is n ≤ 4, too small for statistical significance. The aggregate +16.3% effect (bootstrap CI [+11.0%, +22.6%]) is convincing for the full sample, but the per-model claim is unsupported for the weakest questioners. Please either restrict the behavioral claim to models with sufficient data, or present per-model effect sizes with uncertainty and explicitly note the underpowered subset. This is load-bearing for the abstract's general claim that 'models identify and exploit their peers' cognitive boundaries.'
  3. [§5.2 and §5.3 (Self-Play; Bonus Rounds)] Two supporting analyses lack necessary methodological detail. (1) Self-play: no setup is described in §4—number of self-play games, prompts, temperature, and whether the same judge panel was used. The 93–100% self-answer accuracy is presented as evidence that 'models cannot ask beyond their own knowledge boundary' (Abstract), but without these details it is not auditable. (2) The bonus-round analysis re-classifies 561 bonus questions using an LLM classifier (T=0), but the classifier's accuracy is never reported against a human-labeled sample; errors in dimension assignment could drive the 41.4% vs 18.2% targeting difference. Please add the missing self-play setup and report classifier accuracy on a subsample.
minor comments (5)
  1. [Abstract; §5.1; Limitations] The phrase 'stably separates near-saturated systems' overstates the resolution. Table 3 shows overlapping 95% CIs for adjacent models (e.g., Gemini-3.5-Flash 1642 ± 37 and Gemini-3.1-Pro 1637 ± 40), and the Limitations section acknowledges that 8 games per pair cannot resolve exact order within a tier. Suggest rephrasing to 'resolves five tiers' rather than implying full ordering.
  2. [§5.1, Table 4] GPT-5.5 and Gemini-3.5-Flash both report 100% answer accuracy. Given the small number of rounds per pair, this is striking. Please report the raw number of questions answered by each model and, if possible, provide the distribution of judge-grade margins; otherwise the binary grading may appear artificially lenient.
  3. [§5.3, Figure 5] The bonus-round analysis uses an LLM classifier (T=0) for dimension re-classification but does not report its agreement with the original judge-assigned dimensions or with human labels. A brief calibration table would strengthen the claim.
  4. [§4, Table 2] The 'Context Window' column mixes input and output context lengths without a legend outside the table; clarify the notation in the caption.
  5. [Throughout] The text refers to RQ1, RQ2, RQ3, RQ7, but no formal list of research questions is given. Add a list or rename the references for readability.

Circularity Check

1 steps flagged · score 4.0 of 10

Ground truth is endogenous: the judge panel that defines 'correct' is drawn from the evaluated models, so the 'objective' leaderboard is partly a measure of panel consensus.

  1. self definitional [Abstract; §3 Steps 2 and 4; §4 Experimental Setup; §5.4; Limitations]
    "To ensure questions contain objectively verifiable answers, a judge panel of strong models validates them... The judge panel comprises three of the contestants: GPT-5.5, Gemini-3.5-Flash, and Claude-Sonnet-4.6... Grading is by majority: ≥2 CORRECT votes make the answer correct (gt = 1), otherwise incorrect (gt = 0)... Limitations: a blind spot shared by all of them cannot be probed."

    The only notion of 'correct' is the judge panel's majority vote, and the panel consists of contestants under evaluation. Hence answer accuracy, hit rates, self-harm, and Elo measure agreement with the contestant-judge panel, not independently verified facts. The conclusion that peer probing 'measures factual rigor' and yields an 'objective' leaderboard therefore renames panel consensus as factual rigor by construction. The self-bias audit (Fleiss' κ=0.915) only checks judges against each other, so a shared error is invisible. The limitation that 'a blind spot shared by all of them cannot be probed' applies equally to gold-answer validity: a wrong gold accepted by all judges is treated as true, and a correct answer contradicting it is graded WRONG.

full rationale

The Elo computation itself is not fitted, and no parameter is derived from the target rankings, so this is not a fitted-input-called-prediction case. No load-bearing self-citation chain appears; the cited works are not by the present authors. The circularity is localized to the ground-truth definition: correctness is operationalized as the majority vote of a judge panel that consists of three of the ten evaluated models, and no sample of gold answers or grades is checked against humans or an external benchmark. This makes the 'objective/verifiable' claim partially self-referential. However, the behavioral findings (exploit-after-hit, bonus-round targeting, self-play ceiling) are grounded in tournament logs and are not merely a re-derivation of the leaderboard, and the comparison with human preference provides some external context. Hence partial circularity, not full definitional collapse.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The framework's leaderboard is an observed outcome rather than a fit, but several design choices (K, decay schedule, match structure, judge panel) shape what the Elo values mean. The central empirical result additionally rests on the judge-panel axiom that LLM validation and grading are objectively correct.

free parameters (4)
  • Elo K-factor = 32
    K=32 in Eq. 4; a standard but arbitrary update gain that affects rating spread and sensitivity.
  • Per-dimension decay weights = 1.0 / 0.5 / 0.2 for C<2 / C=2 / C≥3
    §3 Scoring: chosen by the authors to incentivize exploration; directly changes per-round rewards and therefore Elo.
  • Match structure = T=10 rounds, 8 games per pair, 3 judges
    Design choices that set sample size and influence bootstrap CI width; not fitted to data.
  • Judge panel composition = GPT-5.5, Gemini-3.5-Flash, Claude-Sonnet-4.6
    Choice of referees can bias outcomes; the authors audit self-bias but the set remains a free design decision.
assumptions (4)
  • domain assumption LLM judges can determine objective correctness of arbitrary generated questions and answers
    The entire ground truth is the judge panel; §§3, 5.4 report internal agreement but no external validation.
  • domain assumption Dynamically generated questions are contamination-resistant
    Assumes models cannot have memorized newly generated items; plausible but not proven (§1, Table 1).
  • standard math Elo model assumptions: logistic expected score and K-updates are appropriate for this scoring function
    Eqs. 3-4 follow the standard Elo model; relevant only to ranking construction.
  • domain assumption The six capability dimensions cover the relevant skills and are judged consistently
    §3 Capability Dimensions: categories are determined by judge consensus, and the set is a modeling choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LivingArena: Do LLMs Know What Other LLMs Don't? Peer-Probing as Scalable Evaluation." pith.science (2026). https://pith.science/paper/C736ZUBG

@misc{pith2026260724780,
  author       = {Pith},
  title        = {Pith review of: LivingArena: Do LLMs Know What Other LLMs Don't? Peer-Probing as Scalable Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C736ZUBG}},
  note         = {Machine review of arXiv:2607.24780}
}
read the original abstract

Evaluating frontier LLMs is challenging: static benchmarks suffer from contamination and saturation -- leaving users unable to distinguish top models and developers blind to specific failure modes -- while human preference is subjective. In this paper, our question is: \emph{Do LLMs know what other LLMs don't? And can we leverage this dynamic for evaluation?} We present \textbf{LivingArena}, an automated, contamination-resistant evaluation framework. In this framework, models take turns proposing questions, aiming to pose items that opponents cannot answer correctly. Questioners are encouraged to actively identify and exploit opponents' knowledge boundaries, receiving rewards when the answerer fails, while the answerer is rewarded otherwise. To ensure questions contain objectively verifiable answers, a judge panel of strong models validates them, penalizing questioners if validation fails. Evaluating ten frontier LLMs, LivingArena yields a stable Elo leaderboard. Our behavioral analyses show that models identify and exploit their peers' cognitive boundaries: self-play and tournament logs indicate that they localize and double down on opponents' weak dimensions. Beyond static knowledge recall, peer probing measures factual rigor and the higher-order ability to probe an opponent's weaknesses, correlating only weakly with human preference and offering a scalable, low-cost approach to continuous evaluation.

Figures

Figures reproduced from arXiv: 2607.24780 by the authors.

Figure 1
Figure 1. Overview of the LivingArena evaluation pro [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The complete step-by-step execution flow of a bidirectional match in the LivingArena framework. In [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Capability map: hit rate as asker (x) vs. an￾swer accuracy as answerer (y). Questioning and answer￾ing ability are distinct axes; the Claude family answers well but rarely stumps opponents. exhibiting perfect balance: it is a flawless answerer (100% accuracy) and an aggressive, highly effec￾tive questioner (26.3% hit rate). Tier 2 models (Gemini-3.5-Flash and Gemini-3.1-Pro) simi￾larly show high calibration and stro… view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Strategic use of unconstrained bonus rounds [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 4
Figure 4. Figure 4: Sustained probing: baseline hit rate vs. condi [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 6
Figure 6. Figure 6: Self-harm rate per model with bootstrap 95% [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Per-dimension answer accuracy for represen [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 1 linked inside Pith

  1. [1]

    International Conference on Learning Representations (ICLR) , year =

    Measuring Massive Multitask Language Understanding , author =. International Conference on Learning Representations (ICLR) , year =

  2. [2]

    2024 , journal =

    MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark , author =. 2024 , journal =

    2024

  3. [3]

    2024 , journal =

    Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference , author =. 2024 , journal =

    2024

  4. [4]

    2023 , journal =

    Judging LLM-as-a-judge with MT-Bench and Chatbot Arena , author =. 2023 , journal =

    2023

  5. [5]

    2023 , journal =

    Holistic Evaluation of Language Models , author =. 2023 , journal =

    2023

  6. [6]

    Irving, Geoffrey and Christiano, Paul and Amodei, Dario , journal =

  7. [7]

    arXiv preprint arXiv:2312.09390 , year =

    Weak-to-Strong Generalization: Eliciting Strong Capabilities with Weak Supervision , author =. arXiv preprint arXiv:2312.09390 , year =

    arXiv

  8. [8]

    Journal of the Royal Statistical Society: Series C (Applied Statistics) , volume =

    Parameter Estimation in Large Dynamic Paired Comparison Experiments , author =. Journal of the Royal Statistical Society: Series C (Applied Statistics) , volume =

Show all 24 references
  1. [9]

    A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives , author =

  2. [10]

    2026 , eprint=

    QuickScope: Certifying Hard Questions in Dynamic LLM Benchmarks , author=. 2026 , eprint=

    2026

  3. [11]

    arXiv preprint arXiv:2602.02589 , year=

    PeerRank: Autonomous LLM Evaluation Through Web-Grounded, Bias-Controlled Peer Review , author=. arXiv preprint arXiv:2602.02589 , year=

  4. [12]

    2025 , url=

    Ning, Kun-Peng and Yang, Shuo and Liu, Yuyang and Yao, Jia-Yu and Liu, Zhenhui and Tian, Yonghong and Song, Yibing and Yuan, Li , booktitle=. 2025 , url=

    2025

  5. [13]

    arXiv preprint arXiv:2510.22593 , year=

    AutoBench: Automating LLM Evaluation through Reciprocal Peer Assessment , author=. arXiv preprint arXiv:2510.22593 , year=

  6. [14]

    2025 , booktitle=

    ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities , author=. 2025 , booktitle=

    2025

  7. [15]

    , author =

    How Much Do Large Language Model Cheat on Evaluation? Benchmarking Overestimation Under the One-Time-Pad-Based Framework. , author =. 2026 , journal =

    2026

  8. [16]

    2025 , journal =

    SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks , author =. 2025 , journal =

    2025

  9. [17]

    2026 , howpublished=

    BLOOMQA: AUTOMATED BENCHMARK GENERATION FROM DOMAIN GUIDELINES INFORMED BY BLOOM'S TAXONOMY , author=. 2026 , howpublished=

    2026

  10. [18]

    2026 , journal =

    A Survey on Agent-as-a-Judge , author =. 2026 , journal =

    2026

  11. [19]

    2026 , journal =

    Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines , author =. 2026 , journal =

    2026

  12. [20]

    Debate, Deliberate, Decide (D3): A Cost-Aware Adversarial Framework for Reliable and Interpretable

    Harrasse, Abir and Bandi, Chaithanya and Bandi, Hari , booktitle=. Debate, Deliberate, Decide (D3): A Cost-Aware Adversarial Framework for Reliable and Interpretable. 2026 , doi=

    2026

  13. [21]

    2026 , journal =

    Claim Verification with Adversarial Reasoning and Planning , author =. 2026 , journal =

    2026

  14. [22]

    Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval , year=

    PeerPrism: Peer Evaluation Expertise vs Review-writing AI , author=. Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval , year=

  15. [23]

    2026 , eprint=

    Beyond Goodhart's Law: A Dynamic Benchmark for Evaluating Compliance in Multi-Agent Systems , author=. 2026 , eprint=

    2026

  16. [24]

    2025 , eprint=

    CodeArena: A Collective Evaluation Platform for LLM Code Generation , author=. 2025 , eprint=

    2025

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.