{"id":"d3ef4874-3320-49e9-92f4-35db3f53567e","arxiv_id":"2608.18719","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A reference-free LLM judge can separate correct from incorrect answers only when its own competence exceeds the chance floor, and a pre-deployment screen can detect when this fails.","lead":"The paper shows that an LLM judge used to grade AI skill documents is only as good as its own ability to solve the questions, and derives a simple test to screen judges before deployment. A smart generalist would read it to understand when self-grading AI systems can be trusted.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Empirical bound test is underidentified: A1's independence contradicts the paper's own scaffolding account of c_grade > c_solve, so Table 2 cannot confirm the central claim without a direct TPR/FPR decomposition.","rationale":"The paper's formal contribution is clean: Proposition 1 is a simple algebraic consequence of (A1)-(A3), and Corollary 1's necessary condition survives the relaxed collusion model in Remark 1. The difficulty-stratification argument (Remark 2, Proposition 2) is a useful identification correction, and the non-intervening probe is a sound design for measuring discriminability on the verifier-gated distribution. The authors also disclose several limitations (single gold answer, no open-ended tasks, 2-3 seeds), which counts in their favor. My concern is about whether the headline empirical claim, that discriminability is bounded by competence, is actually tested. A1 assumes \\hat a is independent of a; the paper's Section 3.5 scaffolding explanation for c_grade > c_solve is a dependence of \\hat a on a. These cannot both hold. The Table 2 procedure compares observed AUC to U(c_solve). Since U is increasing and c_solve is allowed to be an underestimate, the inequality obs <= U(c_solve) is stronger than the model's actual prediction; it can be satisfied by noise, collusion, or exact-match string effects. The only violation is absorbed by the c_grade gap whose existence would violate A1. The data already contain the information to resolve this: empirical TPR and FPR. If TPR approximately equals c_solve and FPR approximately equals (1-c_solve)/(k-1), the model is the right description and the conditional verdict stands. If TPR exceeds c_solve for correct candidates, scaffolding is real and the bound test is measuring something else. Given the medium risk and the absence of confidence intervals in Tables 1-4, I would keep the verdict CONDITIONAL, with the condition being a direct TPR/FPR decomposition and, ideally, a direct measurement of grading-context competence.","tokens_in":1081,"tokens_out":1036,"duration_ms":140281,"concrete_test":"Re-analyze the stored probe episodes from Section 3.3 and report, per task and judge, the empirical TPR = Pr(s=1|y=1) and FPR = Pr(s=1|y=0) with question-clustered bootstrap intervals, alongside the model's predictions TPR=c_solve and FPR=(1-c_solve)/(k-1). If TPR significantly exceeds c_solve in any cell that admits within-question strata, A1's independence is falsified by scaffolding and Eq. (1) is not the right description of that cell; if FPR significantly exceeds (1-c_solve)/(k-1), collusion is confirmed and only the necessary condition c>1/k remains testable. This uses data already collected in Section 3.3 and directly settles whether the bound is measured or assumed.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The theorem is not the problem; Eq. (1) follows from (A1)-(A3). The problem is that the empirical program cannot simultaneously use (A1) and the paper's own competence-gap story. (A1) in Section 3.1 requires the judge's internal solution \\hat a to be conditionally independent of the candidate answer a; this is what fixes TPR=c and FPR=(1-c)/(k-1). Section 3.5 then explains why closed-book c_solve may understate grading-context c_grade: 'the candidate answer and its reasoning may scaffold re-solving.' That is a direct dependence of \\hat a on a, so (A1) is violated precisely in the regime used to absorb the Table 2 violation (GPQA x Haiku). If A1 fails, Proposition 1's closed form is not the data-generating process and the 'upper bound' is not identified. The tests in Table 2 compare observed AUC to U(c_solve), a lower-bound input to a monotone increasing function; observations below it cannot confirm the model, and the single violation is explained by the same unmeasured c_grade > c_solve gap whose mechanism contradicts A1. What is missing is any direct estimate of the two rates that actually carry the model: TPR and FPR. The screen may still be conservative in practice, but the stated empirical support for 'evaluation bounded by solving' is not currently established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a model of reference-free LLM judges as latent solvers in which a judge's score equals agreement between the candidate answer and the judge's own internal solution. Under assumptions (A1)–(A3), it derives a closed-form expression for ROC-AUC in terms of judge competence c and answer-space size k, a necessary condition c > 1/k for above-chance discrimination, and shows that the marginal AUC is inflated by item-difficulty confounding while a within-question estimator is invariant. It introduces a non-intervening probe that records judge scores on genuine skill-optimization runs without affecting accept/reject decisions, and measures competence via closed-book self-solve, a decontaminated proxy benchmark, and an AUC-inversion estimator. Experiments on three tasks with several judge models find discriminability near chance when competence is near the floor, a bound that holds in stratified cells but is violated in one unstratified cell, and a decomposition of the gap between benchmark accuracy and genuine competence. A small closed-loop study links screening outcomes to false-accept and false-reject rates.","tokens_in":12086,"tokens_out":7606,"duration_ms":66086,"significance":"The theoretical derivation is clean and correct under the stated assumptions, and the diagnostic is practically appealing because it is cheap and non-intervening. The within-question stratification result is a useful identification contribution, and the paper is admirably explicit about its own limitations, including the vacuousness of the AUC-inversion check and the small closed-loop sample. However, the empirical validation is weaker than the abstract suggests: the bound test is one-directional, the one violation is explained by quantities outside the identifying assumptions, and the chance-floor collapse on mathematics is not statistically distinguished from a null result without confidence intervals. The paper therefore needs a revision that provides direct rate estimates and uncertainty quantification before the central claim can be considered established.","major_comments":[{"comment":"Assumption (A1) requires the judge's internal solution \\hat a to be conditionally independent of the candidate answer a, which is what fixes TPR=c and FPR=(1-c)/(k-1). Section 3.5 explains the GPQA × Haiku violation by stating that 'the candidate answer and its reasoning may scaffold re-solving', i.e., \\hat a depends on a, so the explanation operates in a regime where (A1) fails and Proposition 1 is not the data-generating process. In the relaxed family of Remark 1 the model only gives AUC = (1/2)(1+c-rho) with unmeasured rho, so the comparison against U(c_solve) in Table 2 does not identify the bound. Please add a direct TPR/FPR decomposition for the judge scores (for example, against the verifier labels on stored episodes) and report whether TPR and FPR individually match the predicted c and (1-c)/(k-1).","section":"3.1, 3.5"},{"comment":"The empirical bound test is one-directional: since c_solve is used as a lower bound on c_grade and U(c) is increasing, observed AUC below U(c_solve) is automatically consistent with the bound, while observed AUC above it is attributed to the unmeasured c_grade gap or to difficulty confounding (Remark 2). Consequently the rows in which the bound 'holds' provide no confirmatory evidence, and the single violation is explained by the same unmeasured quantities that make the test non-falsifiable in the other rows. A falsifiable test needs either an independent estimate of c_grade or a task that admits within-question strata and for which competence is measured without contamination; as it stands, the table supports only the weaker statement that no clean violation was observed.","section":"4.3, Table 2"},{"comment":"The claim that discriminability 'collapses to chance' near the competence floor is not supported with uncertainty quantification. On research mathematics, \\widehat{AUC}_w = 0.489 based on 87 positive and 123 negative episodes (after stratification on items), which is within one standard error of 0.5 under the usual Mann-Whitney variance formula; no confidence intervals are reported for any of the AUC values or competence estimates in Tables 1, 2, 4, or 5. Without question-clustered bootstrap intervals, the math result cannot distinguish between chance-level discrimination and a small but nonzero effect, and the passing/failing classification in Table 5 that relies on intervals excluding 1/2 is unverifiable. Please report CIs for all headline estimates.","section":"4.2, 4.5"}],"minor_comments":[{"comment":"The notation '[AUC' appears to be a rendering artifact for \\widehat{AUC}; please ensure the hat renders consistently throughout.","section":"Tables 1, 5, Figure 4"},{"comment":"State how many items contribute valid positive/negative strata per task, since the effective sample for AUCw may be far smaller than the total episode count.","section":"3.4"},{"comment":"Define 'empty-inclusive' at first use; the term is intuitive but nonstandard.","section":"3.5"},{"comment":"The conclusion states 'evaluation bounded by solving' as a general result; given the identifiability caveats above, consider phrasing it as supported on the tested stratified tasks.","section":"5"},{"comment":"The reference list is missing preprint numbers for some arXiv items (e.g., [Wei et al., 2026]); please standardize the bibliography.","section":"References"},{"comment":"The caption does not explain the 'MCQ chance floor' marker or region; please clarify so the figure is self-contained.","section":"Figure 4"}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the theoretical core is genuinely new and clean: modeling a reference-free judge as a latent solver gives a closed-form AUC = 1/2 + (ck-1)/(2(k-1)) and a necessary condition c > 1/k for any discriminability. The math is simple enough to check in one sitting, and it holds under the stated assumptions. Second, the empirical support for the central bound is weaker than the framing suggests, and the one reported violation is explained with a mechanism that sits awkwardly against the model's own independence assumption.\n\nWhat's good: the difficulty-confound identification is useful and nicely demonstrated. The marginal AUC can be inflated by item difficulty while a within-question estimator is not, and that shows up in the data (the +0.12 confound on factual QA). The non-intervening probe is the right instrument for measuring judge signal without distorting the optimization dynamics. Cross-provider results with four independent model families showing the same collapse near the competence floor are genuinely supportive of the qualitative claim. The paper is also unusually honest: it flags the vacuous AUC-inversion test, the one-directional bound, the small closed-loop pilot, and the censoring issue. That clarity earns real credit.\n\nWhere it softens: the bound test is one-directional in a way that limits what the \"holds\" rows can establish. Since U(c_solve) is a lower-bound input to an increasing function, observations below it are consistent with the model but also with many alternative stories; they don't confirm the specific mechanism. The single violation (GPQA x Haiku) is explained by two mechanisms, one of which, c_grade > c_solve via scaffolding, requires the judge's internal solution to depend on the candidate answer. That violates A1, which is exactly the assumption that fixes TPR and FPR used to derive the bound. The same move that derives the bound cannot also absorb the violation without additional modeling you don't provide. A direct estimate of the judge's TPR and FPR on graded items, compared to self-solve competence, would settle this, and it is conspicuously absent. The closed-loop pilot at 2-3 seeds supports error-type claims, not effect sizes; the paper says as much, which is fine.\n\nNet: the necessary condition c > 1/k is robust, surviving in the relaxed collusion model, and the qualitative empirical pattern backs it. The exact numeric bound is on shakier ground. The citation pattern looks appropriate, with Dawid-Skene as the right ancestor and the skill-optimization literature covered. I'd send this to peer review, with a referee who asks for the rate decomposition and more seeds. Anyone building judge gates or studying LLM-as-judge reliability should read it.","headline":"A clean formal model and a genuinely useful necessary condition, with an empirical test that is honest but weaker than the prose suggests.","tokens_in":12644,"tokens_out":5047,"duration_ms":46029},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A reference-free LLM judge can separate right from wrong answers only if it can solve the task better than chance, and the paper gives a closed-form bound and a cheap pre-deployment screen for testing that condition.","keywords":["LLM-as-judge","reference-free judge","skill optimization","ROC-AUC","competence","discriminability","validation gate","data contamination"],"falsifier":"Take a task with answer-space size $k=4$, closed-book self-solve competence measured cleanly at $c=0.8$, and multiple evaluations of the same questions. If the within-question stratified AUC sits at or below $1/2$ with no collusion mechanism and no difficulty confound, the claimed necessity of $c>1/k$ would be refuted.","tokens_in":11632,"feed_emoji":"🧠","tokens_out":7067,"duration_ms":62394,"temperature":0.7,"pith_summary":"Text-space skill optimization updates a frozen agent by evolving a natural-language skill document and accepting candidates through a validation gate; today those gates require verifiable rewards. The paper asks whether an LLM judge, which sees no gold answer, can stand in for the verifier. It models a reference-free judge as a latent solver and derives a closed-form bound on discriminability ($\\mathrm{AUC}$) in terms of the judge's competence $c$ and answer-space size $k$, together with a necessary condition $c>1/k$ for any discrimination. Empirically, discriminability collapses to chance near that floor, and the bound holds wherever item difficulty can be controlled; a non-intervening probe makes this measurable before deployment. The result is a cheap screen: estimate uncontaminated competence on the target task and decline to gate on judges near the floor.","feed_headline":"Judge gates need competence above chance","feed_subtitle":"A closed-form bound lets you screen an LLM judge before putting it in charge of skill optimization.","key_machinery":"The central object is the judge-as-latent-solver model: the judge's verdict is the agreement indicator $s=\\mathbb{1}[a=\\hat{a}]$, where $\\hat{a}$ is the judge's own internal answer, produced with probability $c$. This converts grading into an implicit re-solving and yields the identity $\\mathrm{AUC}=\\frac{1}{2}+\\frac{ck-1}{2(k-1)}$, making competence separately measurable by closed-book self-solve accuracy. On the measurement side, the load-bearing instrument is a non-intervening probe that records judge scores on genuine optimization episodes without affecting any accept/reject decision, plus within-question stratification that pairs positive and negative episodes of the same question and uses a question-clustered bootstrap for inference.","core_discovery":"Under assumptions (A1)-(A3) — the judge forms an internal solution conditionally independent of the candidate answer with probability $c$ of matching the gold, its errors spread uniformly over the remaining answers, and it scores by agreement — the paper proves $$\\mathrm{AUC}=\\frac{1}{2}\\left(1+c-\\frac{1-c}{k-1}\\right)=\\frac{1}{2}+\\frac{ck-1}{2(k-1)}.$$ Hence $\\mathrm{AUC}>1/2$ if and only if $c>1/k$. The paper further shows that the marginal AUC is systematically inflated by item-difficulty covariation, while the within-question stratified estimator is invariant to difficulty; the difference between the two estimates the size of the confound. It also shows that label-independent noise shrinks AUC toward chance without changing the sign of the condition, and that under correlated-error departures (collusion) the closed form is an attainable upper bound and only the necessary condition $c>1/k$ survives.","pith_inferences":["Beyond the paper, the same closed-form bound could be applied as a static pre-screen on a held-out item set before any optimization runs, because the probe's non-intervention matters mainly for measuring the true episode distribution rather than for the mathematical bound itself.","Beyond the paper, the collusion parameter $\\rho=\\Pr(\\hat{a}=a\\mid y=0)$ suggests a practical pair of measurements: on negative episodes, measure how often the judge agrees with the candidate's specific error; when $\\rho\\ge c$, even a competent judge cannot be trusted to gate.","Beyond the paper, the free-text prediction that errors almost never coincide verbatim implies that judge competence should transfer most directly to discriminability in open-ended regimes; if rubric-driven open-ended judging behaves differently, the difference would show up as a breakdown of this transfer, which is a testable extension the paper leaves for future work."],"forward_implications":["A judge whose self-solve competence sits at or below the chance floor $1/k$ produces near-random or worse-than-random gate scores, so screening it out before deployment is necessary.","Marginal AUC conflates answer-level discrimination with item difficulty; the within-question stratified estimator, not the marginal one, is the quantity that identifies the effect of interest.","Headline benchmark accuracy overstates the competence that governs discriminability, and the overstatement decomposes into censoring of non-convergent episodes, benchmark exposure, and the solve/grade context gap.","In the closed-loop pilot, a judge that passes the screen never imports harmful candidates, while judges that fail it do so at rates up to a coin flip; passing is a prerequisite for safe gating, not a guarantee of competitive final performance.","In the free-text limit $k\\to\\infty$ the bound becomes $\\mathrm{AUC}\\to(1+c)/2$, so the same unit of competence buys the most discriminability when answers are long-form and errors almost never coincide verbatim."],"supporting_citations":[{"why":"Supplies the noisy-annotator model that the judge-as-latent-solver assumption adapts, with the annotator confusion structure realized as an explicit re-solving process.","marker":"[Dawid and Skene, 1979]"},{"why":"Provides the optimizer and its verifier-gated acceptance loop that the non-intervening probe instruments.","marker":"[Yang et al., 2026]"},{"why":"Documents LLM-judge biases and gameability, motivating the need to test whether judge signal is usable before deployment.","marker":"[Zheng et al., 2023]"},{"why":"Is the source of the GPQA-Diamond benchmark used as the graduate-science test task.","marker":"[Rein et al., 2023]"},{"why":"Supplies the contamination-tracing method underlying the exposure component of the competence-gap decomposition.","marker":"[Golchin and Surdeanu, 2024]"},{"why":"Represents judge-based rewards in reinforcement learning, the broader context that the paper transposes into frozen-weight skill optimization.","marker":"[Corbitt et al., 2025]"}],"fun_headline_variants":["Judge gates need competence above 1/answer space","Closed-form AUC bound for LLM judge gates","Screen LLM judges before putting them in charge","Judge's solving skill caps its rating skill","Pre-deployment diagnostic for judge gates"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a reference-free judge grades by comparing the candidate answer to its own independently formed solution, so its grading accuracy is bounded by its solving accuracy; if the candidate answer itself steers the verdict, or judge and candidate share the same wrong tendencies, the exact closed-form bound no longer holds and only the weaker necessary condition $c>1/k$ survives.","fun_headline_variants_meta":{"raw":{"variants":["Judge gates need competence above 1/answer space","Closed-form AUC bound for LLM judge gates","Screen LLM judges before putting them in charge","Judge's solving skill caps its rating skill","Pre-deployment diagnostic for judge gates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000286,"raw_usage":{"total_tokens":1711,"prompt_tokens":1002,"completion_tokens":709,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":618,"completion_tokens_details":{"reasoning_tokens":640}},"tokens_in":618,"tokens_out":709,"duration_ms":6422,"temperature":1.0,"reasoning_tokens":640,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-27T19:50:10.163484+00:00","model_set":{"reader":"deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.0-alltime-deepseek-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Take a task with answer-space size $k=4$, closed-book self-solve competence measured cleanly at $c=0.8$, and multiple evaluations of the same questions. If the within-question stratified AUC sits at or below $1/2$ with no collusion mechanism and no difficulty confound, the claimed necessity of $c>1/k$ would be refuted.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}