{"id":"75c62d99-984a-4ed6-9e55-05daf7775fe5","arxiv_id":"2607.24780","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Large language models can probe each other's weaknesses in a self-running tournament, producing a stable Elo leaderboard without a static question bank.","lead":"LivingArena lets AI models quiz each other: each model asks questions it thinks another model cannot answer, and judge models check the questions and grade the answers. The result is a self-updating leaderboard that separates top models and exposes which weaknesses each one has.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gold-answer validation by the three-model judge panel is never checked against external truth; shared LLM blind spots could corrupt Elo ratings and the probing analysis.","rationale":"The reader's weakest assumption—that the judge panel can objectively validate gold answers and grades—is indeed the most load-bearing point. I have sharpened it to a specific, untested failure mode: the absence of any external check on gold-answer correctness means that shared blind spots among the three LLM judges become the de facto ground truth. This could systematically bias the leaderboard and the behavioral probing results, precisely because the judges and contestants come from the same model generation and are likely to share misconceptions. The paper provides no evidence against this; the self-bias audit is reassuring only for individual favoritism, not for collective error. The open-source code, clear framework, and stability analyses are genuine strengths, but they do not address this external-validity gap. Because the authors could resolve the concern by adding a human-verified subset and re-running the analysis, a conditional verdict remains appropriate—the concern is real but not fatal. I therefore recommend no change to the reader's CONDITIONAL verdict, as the condition (external verification of judge decisions) is exactly what needs to be satisfied.","tokens_in":10666,"tokens_out":5332,"duration_ms":59155,"concrete_test":"Randomly sample 100 validated gold answers and 100 graded answer–gold pairs. Have human experts (or authoritative references) independently verify the gold answers' correctness and the grading decisions, blind to the original verdicts. Measure human–judge agreement. Additionally, include 50 questions where the correct answer contradicts a common LLM misconception; check whether the judge panel validates the asker's (likely wrong) gold answer while rejecting the correct answerer response. If any significant fraction (>5%) of gold answers are found incorrect, or human–judge grading agreement is below 95%, re-run the Elo computation on the human-verified subset and assess leaderboard shifts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that peer-probing under strict validation yields an objective, contamination-resistant leaderboard—rests entirely on the judge panel's ability to verify gold answers. Yet the only verification is by three LLM judges from the same frontier-model generation, and their consensus is treated as ground truth. This creates a concrete failure mode: if the judges share a false belief or systematic misconception, a wrong gold answer is accepted as valid, and an answerer who provides the truly correct answer (one that contradicts the flawed gold) is graded WRONG. The paper's self-bias audit only measures deviation among judges; it cannot detect errors all three judges make together. The reported Fleiss' κ measures agreement, not correctness. The Limitations section even admits an 'evaluator ceiling' where blind spots shared by all participants cannot be probed—but this applies not only to question difficulty, also to the validity of gold answers. Consequently, the Elo leaderboard and the 'exploit-after-hit' results could reflect the judge panel's shared blind spots rather than objective model ability. For instance, GPT-5.2's low answer accuracy (65.3%) might be partly an artifact of its correct answers being rejected when they conflict with the panel's accepted—but wrong—gold answers. No sample of validated questions or grades is checked against humans or an external benchmark, leaving the framework's objectivity premise unvalidated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LivingArena, a fully automated evaluation framework in which frontier LLMs take turns posing questions to each other in a round-robin tournament. Each question must be accompanied by a gold answer and verification logic; a judge panel of three contestant models validates the question and later grades the answerer's response. The authors run 360 matches among ten models, compute an Elo leaderboard, and report behavioral analyses: models 'exploit-after-hit' (they revisit dimensions on which they previously stumped an opponent), and bonus rounds show strategic re-attack of discovered weaknesses. The paper claims the resulting leaderboard is contamination-resistant, self-adaptive, and measures an objective 'adversarial rigor' axis distinct from human preference.","tokens_in":11002,"tokens_out":6829,"duration_ms":63872,"significance":"If the judge-panel validity concern is resolved, LivingArena is a valuable contribution: it is fully automated, requires no static question bank, is scalable (~100 minutes for a ten-model tournament), and the authors release code and raw logs. The distinction between asking and answering ability and the self-harm calibration signal are genuinely useful diagnostics. The exploit-after-hit behavioral finding is interesting and, for stronger questioners, supported by effect sizes with bootstrap confidence intervals. The core risk is that the entire pipeline treats the three-model judge panel's consensus as objective ground truth, and this premise is never externally validated.","major_comments":[{"comment":"The framework's objectivity premise rests on the judge panel (three LLM contestants) validating gold answers and grading responses. The paper reports high inter-judge agreement (Fleiss κ = 0.915 grading, 0.693 validation) and a self-bias audit, but agreement is not correctness. No sample of validated questions or graded responses is checked against human experts or an external benchmark. Concretely: if the three judges share a systematic misconception, a wrong gold answer is accepted, and an answerer giving the truly correct answer (contradicting the flawed gold) is graded WRONG. The Limitations section admits an 'evaluator ceiling' for shared blind spots, but applies it only to probing difficulty, not to gold-answer validity. I recommend a human-expert audit of a random sample (e.g., 200 questions across dimensions) to estimate the judge panel's error rate, and report panel-vs-external","section":"§3 (Steps 2 & 4); §5.4; Limitations"},{"comment":"The paper claims the exploit-after-hit effect is 'positive for all ten models' and that models 'double down on discovered vulnerabilities.' However, §5.3 states that for the Claude family and GPT-5.2 (weak questioners), the sample size of post-first-hit rounds is n ≤ 4, too small for statistical significance. The aggregate +16.3% effect (bootstrap CI [+11.0%, +22.6%]) is convincing for the full sample, but the per-model claim is unsupported for the weakest questioners. Please either restrict the behavioral claim to models with sufficient data, or present per-model effect sizes with uncertainty and explicitly note the underpowered subset. This is load-bearing for the abstract's general claim that 'models identify and exploit their peers' cognitive boundaries.'","section":"§5.2 and §5.3 (Exploit-after-Hit, Sustained Probing)"},{"comment":"Two supporting analyses lack necessary methodological detail. (1) Self-play: no setup is described in §4—number of self-play games, prompts, temperature, and whether the same judge panel was used. The 93–100% self-answer accuracy is presented as evidence that 'models cannot ask beyond their own knowledge boundary' (Abstract), but without these details it is not auditable. (2) The bonus-round analysis re-classifies 561 bonus questions using an LLM classifier (T=0), but the classifier's accuracy is never reported against a human-labeled sample; errors in dimension assignment could drive the 41.4% vs 18.2% targeting difference. Please add the missing self-play setup and report classifier accuracy on a subsample.","section":"§5.2 and §5.3 (Self-Play; Bonus Rounds)"}],"minor_comments":[{"comment":"The phrase 'stably separates near-saturated systems' overstates the resolution. Table 3 shows overlapping 95% CIs for adjacent models (e.g., Gemini-3.5-Flash 1642 ± 37 and Gemini-3.1-Pro 1637 ± 40), and the Limitations section acknowledges that 8 games per pair cannot resolve exact order within a tier. Suggest rephrasing to 'resolves five tiers' rather than implying full ordering.","section":"Abstract; §5.1; Limitations"},{"comment":"GPT-5.5 and Gemini-3.5-Flash both report 100% answer accuracy. Given the small number of rounds per pair, this is striking. Please report the raw number of questions answered by each model and, if possible, provide the distribution of judge-grade margins; otherwise the binary grading may appear artificially lenient.","section":"§5.1, Table 4"},{"comment":"The bonus-round analysis uses an LLM classifier (T=0) for dimension re-classification but does not report its agreement with the original judge-assigned dimensions or with human labels. A brief calibration table would strengthen the claim.","section":"§5.3, Figure 5"},{"comment":"The 'Context Window' column mixes input and output context lengths without a legend outside the table; clarify the notation in the caption.","section":"§4, Table 2"},{"comment":"The text refers to RQ1, RQ2, RQ3, RQ7, but no formal list of research questions is given. Add a list or rename the references for readability.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is well written and the open-source release is a strong asset. My primary concern—unvalidated judge-panel ground truth—is, in my view, serious enough to require a human-expert audit before the central claims can be accepted. The per-model exploit-after-hit overclaim and missing methodological details for self-play and the bonus classifier are secondary but should be addressed in the same revision. I encourage the editor to send the paper back for major revision rather than reject; the architecture and core behavioral effect are promising if the validity gap can be closed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real contribution, not a stunt. The idea of having models compose verification questions for each other and score them in an Elo tournament is well-specified, shipped as code, and produces internally coherent behavioral results that static benchmarks would not surface. The exploit-after-hit pattern (+16.3pp), bonus-round re-attack (41.4% vs 18.2%), and self-harm calibration splits are the most interesting findings. The self-bias audit is the right check, and the leaderboard is not fitted to itself, so \"circular\" is the wrong word.\n\nWhere I share the skeptic's worry: the whole pipeline depends on three LLM judges being able to verify gold answers and grade responses. The paper measures inter-judge agreement—which is consistency, not correctness—and controls for self-preference by item difficulty, which is good. But it never checks a sample of validated questions or grades against human experts or an external benchmark. If all three judges share a blind spot, a wrong gold passes validation and a correct answer gets marked wrong. The paper's own limitation section admits the shared-blind-spot problem for probing difficulty, and the same logic applies to gold validity. So the objective-truth premise is plausible but unvalidated. The 100% answer accuracy for GPT-5.5 and Gemini-3.5-Flash reinforces that concern; it makes me want to see a human-reviewed subset before trusting the point estimates.\n\nMinor issues: post-first-hit analyses for weak questioners have n≤4 and are appropriately labeled; leaderboard CIs within tiers overlap, so tier boundaries are fine but within-tier ordering is not; 8 games per pair is thin for fine ranking. These are not fatal, but they limit what the leaderboard can support.\n\nWho benefits: people building evaluation harnesses for frontier models will take this as a strong baseline idea, and the open source code and logs are a real plus. The paper deserves a serious referee. If I were editing, I would send it out and ask for one additional section: a human-verified or externally grounded sample of judge decisions, plus a same-model comparison to a static benchmark to show the axis is distinct. That would turn a promising framework into a convincing one.","headline":"A real and well-specified evaluation idea—peer models generating, validating, and grading questions in an Elo tournament—worth refereeing, but the objectivity claim rests on unvalidated judge consensus.","tokens_in":11437,"tokens_out":1741,"would_cite":true,"duration_ms":18484,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By letting frontier LLMs interrogate one another with judge-verified, freshly generated questions, LivingArena produces a stable Elo leaderboard—and shows that the models genuinely find and re-attack their peers' weak dimensions.","keywords":["LLM evaluation","peer probing","adversarial tournament","Elo rating","benchmark contamination","self-calibration","LLM-as-a-judge","knowledge boundaries"],"falsifier":"Sample, say, 200 questions and 200 graded responses from the tournament logs; have independent human experts—or executable checkers for code and math items—re-verify the gold answers and re-grade the responses, then compare with the judge panel's votes. If disagreement is large or correlates with the identity of the asker or answerer, the claimed objective-verification guarantee fails and the ranking cannot be trusted as a measure of ability.","tokens_in":10592,"feed_emoji":"⚔️","tokens_out":6317,"duration_ms":59939,"temperature":0.7,"pith_summary":"LivingArena is a proposal to replace static, human-written benchmarks with a zero-sum game in which frontier LLMs take turns asking each other questions that a panel of model judges must verify as objective and must grade as correct or wrong. The paper's central claim is that this peer-probing game yields a stable Elo leaderboard that separates models that static benchmarks can no longer distinguish, and that the separation is not an artifact: tournament logs show models localizing and re-attacking an opponent's weak dimension after a successful hit. If the claim holds, LLM evaluation can become self-adapting and contamination-resistant without human authoring, and it measures a distinct axis—objective rigor, factual calibration, and the higher-order ability to probe a peer's blind spots—that human preference voting does not capture. The framework also exposes practical diagnostics such as self-harm rate, hit rate as asker, and answer accuracy, which point to specific failure modes.","feed_headline":"Peer-probing tournament ranks top LLMs in a stable Elo order","feed_subtitle":"Freshly generated, judge-verified questions dodge benchmark contamination and expose weak spots static tests miss.","key_machinery":"The central object is LivingArena's bidirectional adversarial match: ten rounds of question-generation and answering in each direction (A asks B, then B asks A). The asker must submit a question, a gold answer, and verification logic; a panel of three contestant models vetoes any question that any judge finds invalid (self-harm penalty −1.0), the answerer is graded by majority CORRECT/WRONG, and successful hits are decay-weighted per dimension (1.0, 0.5, 0.2) to encourage breadth. Net scores map to a pairwise outcome, Elo is updated with K=32, and six capability dimensions give per-dimension diagnostics. The mechanism turns 'a question you cannot answer' into a verifiable fact/reasoning-trap","core_discovery":"On the paper's own terms, the discovery is twofold. First, peer probing works as an evaluation: ten frontier models played a round-robin tournament of 360 matches (3,600 rounds), and the resulting Elo ratings separate five tiers with bootstrap-stable ranks, even where static benchmarks saturate. Second, the tournament logs show the mechanism is real: models answer their own validated questions in 93–100% of self-play rounds, yet re-target the same dimension after a hit 20.1% of the time versus 3.8% after a miss, and in unconstrained bonus rounds they attack a previously-hit weak dimension 41.4% versus 18.2% for never-hit dimensions, with 70.7% of bonus questions landing in discovered weak ar","pith_inferences":["Editorial extension: because the judges are themselves contestants, the framework's gate is the panel's reliability; an easy robustness check is to rerun the tournament with an independent, less capable judge set and see whether the Elo order changes.","Editorial extension: the paper's stated 'shared blind spot' limit means peer-probing cannot detect gaps common to all participants; injecting external verifiers (e.g., program execution, symbolic solvers, or retrieved sources) for selected dimensions would extend it beyond pure self-reference.","Editorial extension: the exploit-after-hit effect suggests a predictive use: before release, a model could be run against a small panel of peers, and the dimensions where they concentrate attacks could serve as a list of likely failure modes—testable against held-out human error data."],"forward_implications":["Because questions are generated fresh each round, the approach resists contamination from fixed question banks and can regenerate difficulty as models improve.","Asking and answering are separable axes; a two-axis capability map (hit rate vs. accuracy) reveals whether a model's deficit is knowledge, questioning strategy, or calibration.","Self-harm rate becomes a calibration diagnostic: a high rate identifies models that often produce incorrect or unverifiable gold answers, a failure mode that aggregate accuracy hides.","A full ten-model round-robin tournament runs in about 100 minutes with parallel API calls, making continuous, low-cost regression evaluation feasible without human authoring.","The per-dimension scoring and bonus-round logs can localize which capability dimension drove a model's rank, rather than giving only a single aggregate number."],"fun_headline_variants":["LLMs probe each other's weak spots in LivingArena tournament","Peer-probing beats static benchmarks, ranks LLMs via Elo","Models expose peer limits: LivingArena's scalable eval","Tournament of questions: LLMs rank each other reliably","Do LLMs know each other's gaps? LivingArena says yes"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"Around §3 and §5.4, the whole leaderboard rests on the assumption that three LLM contestants acting as judges can always decide whether a proposed question has a correct gold answer and whether a response matches it; if validation or grading errors are frequent or biased toward a model's family, Elo would rank judge preferences rather than objective ability.","fun_headline_variants_meta":{"raw":{"variants":["LLMs probe each other's weak spots in LivingArena tournament","Peer-probing beats static benchmarks, ranks LLMs via Elo","Models expose peer limits: LivingArena's scalable eval","Tournament of questions: LLMs rank each other reliably","Do LLMs know each other's gaps? LivingArena says yes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000175,"raw_usage":{"total_tokens":1143,"prompt_tokens":782,"completion_tokens":361,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":273}},"tokens_in":526,"tokens_out":361,"duration_ms":4158,"temperature":1.0,"reasoning_tokens":273,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T10:39:58.722290+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sample, say, 200 questions and 200 graded responses from the tournament logs; have independent human experts—or executable checkers for code and math items—re-verify the gold answers and re-grade the responses, then compare with the judge panel's votes. If disagreement is large or correlates with the identity of the asker or answerer, the claimed objective-verification guarantee fails and the ranking cannot be trusted as a measure of ability.","supporting_citations":[],"review_version":1}