Pith. sign in

REVIEW 4 major objections 8 minor 16 references

Measure AI by the variance its questions induce across solvers

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-09 21:07 UTC pith:VWYG3HYR

load-bearing objection SepaRank is a genuinely novel evaluation framework, but the empirical validation has a conspicuous gap: the one arm that would isolate genuine capability from strategic exploitation is described but never reported. the 4 major comments →

arxiv 2607.07040 v1 pith:VWYG3HYR submitted 2026-07-08 cs.AI

Measuring Intelligence Beyond Human Scale

Jerry Han , Rafael Moschopoulos , Ella Colby , Vishrut Goyal , Andrew Tu , Kia Ghods , Mark Braverman , Elad Hazan This is my paper
classification cs.AI
keywords adversarial psychometricsrelative measurementmodel-generated evaluationseparative measurementvariance-based scoringjudge-free adjudicationbenchmark saturationBrier loss
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that absolute-scale evaluation — where humans design tasks and score models against fixed benchmarks — cannot scale beyond human capability, because examiners eventually lack the knowledge to formulate tasks that are both hard and verifiable. The proposed alternative is adversarial psychometrics: models generate binary challenges and are rewarded not for defeating a single opponent, but for the variance they induce across a population of solvers' reported probabilities. This shift from pairwise to separative measurement is the paper's central move. The authors claim it removes the incentive for private-information and trapdoor attacks — a question exploiting hidden knowledge about one solver produces no variance across a population of calibrated solvers — and yields a rating system, SepaRank, that scales with the systems being measured and can operate without a human judge.

Core claim

The paper's central technical claim is that the variance of solver-reported probabilities across a population is a robust proxy for the discriminating power of a challenge, and that aggregating proposer rewards (for inducing variance, scored as 4·Var(p)) with solver losses (Brier score against a resolved answer) produces a stable ranking. The key insight is structural: a private-state question like 'which bit am I thinking of?' earns zero score because calibrated solvers all report 1/2, so the variance objective automatically penalizes information that only the proposer holds. Empirically, across program (verifiable) and question (non-verifiable) arms with eleven frontier models, the two arm

What carries the argument

The protocol's engine is the proposer reward 4·Var(p), where p_i is each solver's reported probability that the answer is 1. This single formula replaces pairwise win/loss with population-level separation: a challenge scores well only if it splits the field. Solvers are scored by Brier loss (p−y)² against a resolved bit y, which can be determined by program execution, proposer commitment, or panel consensus. Per-round scores standardize both roles as z-scores and sum them; adaptive multiplicative-weights updating shifts sampling toward the frontier over time.

Load-bearing premise

The protocol assumes that inducing variance across a population of solvers measures genuine intelligence gaps rather than shared blind spots or trivial exploits, and that solvers are rational enough to punish deception by hedging — conditions the paper itself notes are not guaranteed for current models.

What would settle it

Run SepaRank on a population of solvers that share a known systematic bias (e.g., all trained on the same contaminated data) and check whether the ranking tracks the bias rather than capability; if variance rewards exploit shared artifacts, the ranking is invalid as an intelligence measure.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Evaluation infrastructure could become self-scaling: as models improve, the benchmark automatically generates harder challenges without human authoring or curation.
  • Judge-free resolution (proposer-committed or consensus) opens evaluation in domains where ground truth is expensive, subjective, or impossible to verify externally.
  • The protocol doubles as a data-generation pipeline: the challenges that separate models are exactly the training signal needed for targeted capability improvement.
  • If adopted, the separative formulation could replace static leaderboards with continuously evolving, population-relative rankings that track the frontier as it moves.
  • The distinction between verifiable and committed resolution rules suggests a spectrum of evaluation strictness that can be matched to domain cost.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The protocol implicitly assumes sufficient solver diversity; if all solvers share the same training-data artifacts, variance could measure shared blind spots rather than genuine capability gaps, and the ranking would track exploitability rather than intelligence.
  • The committed-resolution variant creates a game-theoretic equilibrium where honesty is rewarded only if solvers are rational enough to punish deception by hedging — a condition that may not hold for current models, and the paper itself notes no guarantee that competitors are rational.
  • A natural validation test would be whether SepaRank rankings predict transfer performance on held-out human-authored tasks; if separation ability does not correlate with standard capability measures, the proxy is suspect.
  • The variance objective could be replaced by structured separation objectives (the paper mentions Jensen-Shannon divergence) that might better capture multi-modal or non-monotonic capability differences across a heterogeneous solver population.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper proposes SepaRank, an adversarial psychometric framework for evaluating AI systems beyond human-scale capability. The core idea is that instead of pairwise comparison (which is vulnerable to private-information and trapdoor attacks), a proposer model submits a binary challenge to a panel of solvers and is rewarded for inducing variance in their reported probabilities. Solvers are scored by a proper scoring rule (Brier loss) against a resolved answer. The framework includes adaptive weighting (Section 3.3) to focus measurement on the current frontier, and supports multiple resolution rules (execute, committed, consensus). The authors instantiate the protocol with 11 contemporary models across two arms (program/committed and question/committed) and demonstrate that the protocol produces rankings, that frontier models separate from the field, and that rational reasoning can defeat exploits in the open-protocol setting.

Significance. The paper addresses a genuine and timely problem: the saturation of human-authored benchmarks and the difficulty of constructing verifiable, discriminating tasks above human expert level. The central move from pairwise to separative (one-to-many) measurement is conceptually clean and directly addresses the trapdoor/private-information failure mode identified in Section 2.1. The framework is falsifiable: the scoring formulas are fully specified, the protocol parameters are listed (Table 6), and the experimental instantiation with real models demonstrates the mechanism runs end-to-end. The observation that opus-4.8 fails in the program arm due to cooperative rather than adversarial authoring style (Section 4.3) is a genuinely interesting finding about the relationship between authoring strategy and measured capability. The connection to multi-prover interactive proofs (Section 1.1), while informal, provides useful conceptual grounding.

major comments (4)
  1. The paper claims to 'support judge-free adjudication' (Abstract) and describes four arms (PE, PC, QC, QN in Table 7 and Section 3.4), but reports experimental results for only PC and QC (Section 4.1). Both reported arms use committed resolution, where the resolved answer is the proposer's committed bit rather than an objective truth. The PE arm (program/execute), where the resolved answer is the program's actual output and dishonest commitment is impossible, is the one condition that would test whether the PC rankings reflect genuine capability rather than strategic sophistication. Its absence is conspicuous because the paper's own evidence shows that in the committed arms, dishonest commitment pays 2.4x more than honesty (mean proposer reward 0.36 vs 0.15, Section 4.4), and the PC winner gpt-5.5 is the least honest proposer at 59% honesty. The paper argues this self-corrects in the long
  2. Section 3.3 (adaptive weighting): The weights are updated based on past scores (Eq. for w^(r+1)), and these weights determine panel sampling, which in turn affects future scores. This creates a feedback loop that could amplify early advantages or disadvantages. The paper does not provide sensitivity analysis on the key parameters (eta=0.2, epsilon=0.05, update cadence=5 rounds, z_min=0.02). Since the final ranking G is the sum of round scores g, and g depends on z-scored quantities that depend on which solvers are sampled (which depends on weights), the ranking is path-dependent. The paper should either (a) report the uniform-sampling ablation (mentioned as implemented in Appendix D but never used in the main results) to show the adaptive weighting does not materially change rankings, or (b) provide justification for the specific parameter choices.
  3. The load-bearing assumption that 'separation' reflects genuine capability rather than shared blind spots or training-data artifacts is not independently validated. The paper notes (Section 4.5) that without reasoning, models are vulnerable to exploits, and relies on the assumption that 'the entire field is sufficiently rational' to defeat them. The reasoning experiment (Section 4.5, Figure 3) shows only moderate agreement with the no-reasoning arm (Spearman rho=0.70), and several models shift rank substantially (Haiku declines, Kimi and DeepSeek gain). This suggests the ranking is sensitive to whether models can reason about the protocol, which is a property of the solver pool rather than a stable property of the systems being measured. The paper should discuss whether the ranking would stabilize as all models become more rational, or whether it is inherently dependent on the specific混合.
  4. Section 4.4 reports that dishonest proposals earn mean proposer reward 0.36 vs 0.15 for honest ones, and that the winner gpt-5.5 is the least honest proposer (59% honesty). The paper argues that 'false commitments may increase disagreement in the short run, they should eventually erode trust' (Section 3.4), but provides no evidence of convergence over the 20 rounds. Figure 2 shows gpt-5.5's score growing approximately linearly, suggesting the exploitation persists throughout. The paper should either (a) report whether solver hedging toward 0.5 increases in later rounds for dishonest proposers, or (b) acknowledge that 20 rounds may be insufficient for the self-correction mechanism to engage, which limits the judge-free claim.
minor comments (8)
  1. Section 2.1: The trapdoor factoring example is effective, but the transition from 'private-state question' to 'algorithmic backdoor' could be stated more explicitly as both being instances of asymmetric information.
  2. Table 2: The 'Honesty' column is defined in the caption as 'Pr(committed = executed) among valid proposals' but this definition only appears in the table caption, not in the main text where honesty is discussed (Section 4.4). Cross-referencing the table definition in Section 4.4 would help.
  3. Section 4.3: The observation that opus-4.8's proposals carry comments 92% of the time (vs 3.5% for gpt-5.4) is interesting but the causal claim that this is why its proposals fail to divide the panel should be qualified. An alternative explanation is that opus-4.8 selects easier problems; the comment rate may be a correlate rather than a cause.
  4. Figure 1: The diagonal line is labeled 'G_QC = G_PC' in the caption but the axis labels use 'total score G, program/committed' and 'total score G, question/committed'. Consistent labeling would help.
  5. Section 4.5, Table 4: The table reports 'Total explicit-reasoning characters written' but does not explain why this metric is relevant. A brief justification for why scratchpad length matters for the analysis would help.
  6. Appendix A, Table 5: The representative questions are labeled as 'selected for interest, not for frequency' but several (e.g., identity-based, self-referential) are discussed as strategic patterns in Section 4.5. Clarifying whether these patterns are common or rare would help readers calibrate the prevalence of exploits.
  7. The paper uses model names (gpt-5.5, opus-4.8, etc.) that are not publicly available at the time of review. This is understandable for a forward-looking paper, but it limits reproducibility. A note acknowledging this would be appropriate.
  8. Section 3.2: The probability amplification mitigation (rounding to {0, 1/2, 1}) is described but noted as not applied in the experiments. A brief note on whether probability amplification was observed in the data would help readers assess whether this mitigation is needed.

Circularity Check

0 steps flagged

No circularity: SepaRank's scoring mechanism is self-contained and independently defined

full rationale

The paper's derivation chain is self-contained. The proposer reward (4 Var(p), Section 3) and solver loss (Brier loss, Section 3.1) are defined independently — variance in solver reports is not defined in terms of the rating it produces, and Brier loss is a standard proper scoring rule. The adaptive weighting mechanism (Section 3.3) updates weights based on past scores and uses them for panel sampling, but the paper explicitly states weights 'serve only to bias panel sampling toward stronger solvers; they play no role in the rating itself' — the final ranking is the sum of round scores G_i, not the weights. The round score g_i = 0.5*z(P) + 0.5*z(Q) standardizes proposer and solver quantities within each round, which is a normalization, not a circular definition. The paper cites prior psychometric work (Spearman 1904, Rasch 1960) and related work (MathDuels, Xu et al. 2026) for context, but these are not load-bearing for the mathematical framework — the scoring formulas are derived from first principles within the paper. The experimental results (Section 4) test the protocol against external model populations and report empirical outcomes (honesty rates, separation scores) that are not forced by construction. While the skeptic raises valid concerns about whether the committed-resolution arms measure strategic sophistication rather than intelligence, and the absence of PE arm results is conspicuous, these are correctness/validity concerns rather than circularity — the protocol's outputs are not equivalent to its inputs by definition.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 2 invented entities

The axiom ledger captures the key assumptions and parameters of the SepaRank protocol. The free parameters are implementation choices for the adaptive weighting and scoring mechanisms. The axioms include inherited psychometric assumptions and new assumptions specific to the adversarial framework. The invented entities are the protocol and paradigm themselves, which are validated empirically.

free parameters (5)
  • z_min = 0.02
    Floor for z-score divisor to prevent noise in low-spread rounds.
  • eta = 0.2
    Multiplicative-weights adaptation rate for panel sampling.
  • epsilon = 0.05
    Uniform mass mixed into weight updates to prevent zero weights.
  • update_cadence = 5
    Number of rounds between weight updates.
  • history_window = 5
    Number of past rounds shown in model prompts.
axioms (4)
  • domain assumption Intelligence can be measured through performance on tasks rather than defined independently of behavior.
    Inherited from classical psychometrics (Spearman, Rasch); stated in Section 1.
  • ad hoc to paper The ability to induce variance in a population of solvers is a valid proxy for intelligence.
    Central to the SepaRank protocol; the paper does not prove this but relies on it for validity.
  • ad hoc to paper Models are sufficiently rational to defeat exploits in the long run under open protocols.
    Section 4.5 relies on this assumption to argue that the protocol will not collapse under trivial questions.
  • domain assumption Proper scoring rules incentivize calibrated belief reporting in LLMs.
    Used to score solvers; the paper provides empirical evidence that models fail to hedge, suggesting this axiom may not hold perfectly.
invented entities (2)
  • SepaRank independent evidence
    purpose: The novel ranking protocol based on separative measurement.
    Empirically instantiated and tested in Section 4.
  • Adversarial psychometrics independent evidence
    purpose: The new paradigm of evaluating intelligence by the ability to discover capability boundaries.
    The conceptual framework underlying SepaRank.

pith-pipeline@v1.1.0-glm · 24417 in / 2477 out tokens · 291045 ms · 2026-07-09T21:07:50.180907+00:00 · methodology

0 comments
read the original abstract

How can we measure intelligence beyond human capability? Human-authored benchmarks saturate, and above human capability, examiners may not know which tasks are both hard and verifiable. We argue that this difficulty is inherent to absolute-scale evaluation and propose a new paradigm based on relative measurement in which models generate public challenges that separate other systems. Aggregating these outcomes yields an adversarial psychometric rating system that can scale with the systems being measured. We describe practical protocols that reduce incentives for private-information attacks, support judge-free adjudication, and naturally scale with agent capabilities. We instantiate the framework across verifiable and open-ended, non-verifiable domains, illustrating how model-generated evaluation can continue to measure systems beyond the human frontier.

Figures

Figures reproduced from arXiv: 2607.07040 by Andrew Tu, Elad Hazan, Ella Colby, Jerry Han, Kia Ghods, Mark Braverman, Rafael Moschopoulos, Vishrut Goyal.

Figure 1
Figure 1. Figure 1: Total score G in the program arm against the question arm, mean and 95% CI over the ten games of each arm. The dashed line is GQC = GPC. The arms agree on the ordering (Spearman ρ = 0.93); models above the line are relatively stronger on questions. 4.3 Dynamics 5 10 15 20 0 10 round mean cumulative score G (a) PC: program / committed 5 10 15 20 −10 0 round (b) QC: question / committed gpt-5.5 gpt-5.4 qwen3… view at source ↗
Figure 2
Figure 2. Figure 2: Mean cumulative score G by round, averaged over the ten games of each arm. The leaders’ totals grow at a nearly constant per-round rate, while several mid-field models peak early and decay: opus-4.8 in the program arm peaks by round two, and qwen3.7-max and haiku-4.5 trace the same arc in the question arm. The bottom of the field separates within the first five rounds [PITH_FULL_IMAGE:figures/full_fig_p01… view at source ↗
Figure 3
Figure 3. Figure 3: Total score G with an explicit reasoning field against without one, mean and 95% CI over the ten games of each arm (question/committed, author disclosure). The dashed line is Greas = Gnoreas. The arms agree only moderately on the ordering (Spearman ρ = 0.70); models above the line are relatively stronger with the reasoning field. sponses is to simply demand it. Many powerful solvers reasoned that questions… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 16 canonical work pages · 8 internal anchors

  1. [2]

    Concrete Problems in AI Safety

    URLhttps://arxiv.org/ abs/1606.06565. L´ aszl´ o Babai, Lance Fortnow, and Carsten Lund. Non-deterministic exponential time has two-prover interactive protocols.Computational Complexity, 1(1):3–40, Mar

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [4]

    Measuring Progress on Scalable Oversight for Large Language Models

    URLhttps://arxiv.org/abs/2211.03540. 15 Ralph Allan Bradley and Milton E. Terry. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons.Biometrika, 39(3/4):324–345,

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [6]

    On the Measure of Intelligence

    URL https://arxiv.org/abs/1911.01547. Paul Christiano, Buck Shlegeris, and Dario Amodei. Supervising strong learners by amplifying weak experts.arXiv preprint arXiv:1810.08575,

    work page internal anchor Pith review Pith/arXiv arXiv 1911

  4. [7]

    Supervising strong learners by amplifying weak experts

    URLhttps://arxiv.org/abs/1810.08575. Edoardo Debenedetti, Jie Zhang, Mislav Balunovi´ c, Luca Beurer-Kellner, Marc Fischer, and Florian Tram` er. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. InAdvances in Neural Information Processing Systems (NeurIPS), volume 37, pages 82895–82920,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [8]

    doi: 10.52202/079017-2636. Arpad E. Elo.The Rating of Chessplayers, Past and Present. Arco Publishing, New York,

    work page doi:10.52202/079017-2636

  6. [10]

    FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI

    URLhttps://arxiv.org/abs/2411.04872. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding. InInternational Conference on Learning Representations,

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [12]

    URLhttps://arxiv.org/abs/1805.00899. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. Dynabench: Rethinking Benchmar...

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  8. [13]

    Fine-grained prediction of reading comprehension from eye movements,

    doi: 10.18653/v1/ 2021.naacl-main.324. 16 Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic Evaluation of Language Models.Transactions on Machine Learning Research,

    work page doi:10.18653/v1/ 2021

  9. [14]

    Humanity's Last Exam

    ISSN 1476-4687. doi: 10.1038/s41586-025-09962-4. Georg Rasch.Probabilistic Models for Some Intelligence and Attainment Tests. Danish Institute for Educational Research, Copenhagen,

    work page internal anchor Pith review doi:10.1038/s41586-025-09962-4

  10. [16]

    MathDuels: Evaluating LLMs as Problem Posers and Solvers

    URLhttps://arxiv.org/abs/2604.21916. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph Gonzalez, and Ion Stoica. Judging LLM- as-a-Judge with MT-Bench and Chatbot Arena. InAdvances in Neural Information Processing Systems (NeurIPS), volume 36, pages 46595–46623,

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [17]

    Is the city of Windsor, Ontario, Canada lo- cated farther south (at a lower latitude) than the city of Detroit, Michigan, USA?

    17 A Authored challenges: templates, exploits, and strategies Table 5 lists representative high-separation questions verbatim. Table 5: Representative high-separation questions, with the proposer’s committed bitc, the resolved truthy, the panel’s yes-votes, and the proposer rewardP. Selected for interest, not for frequency. Question (verbatim) Armc yyesP ...

    work page 2026

  12. [18]

    strawberry

    Statement B: Erin committed answer 0 to this very ques- tion.” QC 1 1 2/5 0.86 Counterintuitive-but-objective facts.Windsor liessouthof Detroit, and France borders Brazil (through French Guiana). Character- and token-level questions.A second cluster targets the well-documented sub-token blind spot of counting letters in “strawberry” or “MISCELLANEOUS”. We...

    work page 2026

  13. [19]

    Judy”; seed s5 r11; repeated “current calendar year

    Since Erin’s commits are unpredictable 19 and likely chosen to maximize variance. . . To minimize my maximum possible loss (minimax strat- egy). . . I will report a probability of 0.5, guaranteeing a Brier loss of 0.25 regardless of her commit.” Tracking an inconsistent single-fact proposer(Kimi K2.7, playing “Judy”; seed s5 r11; repeated “current calenda...

    work page 2025

  14. [20]

    Grace”; seed s9 r0; “is the 1000th decimal digit ofπodd?

    He has also asked ‘. . . 2026?’ in Round 9, committing answer 1 (which is false in reality but that’s what he committed). . . Given his established pattern of commit- ting 1 for these year questions regardless of actual truth. . . I should report p=1.0 to match Dave’s likely commitment and minimize Brier loss.” Correctly hedging an intractable computation...

    work page 2026

  15. [21]

    - At most 2000 characters

    - The question must be objectively resolvable. - At most 2000 characters. - MACHINE-CHECKED (voids the proposal and scores -1): a reply that does not parse as 24 the JSON below, an empty or over-length question, or a committed answer that is not 0 or

    work page 2000

  16. [22]

    Below is a general yes/no question

    >>>> under committed resolution (arm QC) that sentence instead reads: THIS ROUND YOU ARE A SOLVER. Below is a general yes/no question. Report your probability p that the RESOLVED answer is 1 -- under this run’s rules that is the answer the proposer COMMITTED. Question: <QUESTION> Respond with ONLY a JSON object on one line and nothing else: {"p_one": <num...

    work page 2024