Pith. sign in

REVIEW 2 major objections 3 minor 7 references

This paper shows that observed agreement and disagreement counts among LLM judges can logically certify that at least one judge is failing a stated grading-accuracy requirement, with no false positives and no knowledge of the correct answer

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A no-knowledge alarm can prove, without an answer key, that at least one of several LLM judges fails a user-specified per-label accuracy requirement, by showing every possible ground-truth label assignment is infeasible.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection The LP-feasibility idea is sound but the paper's own running example is false: the alarm as specified cannot fire on the 25 MT-Bench comparisons. the 2 major comments →

arxiv 2509.08593 v1 pith:2O5ASXQB submitted 2025-09-10 cs.AI stat.ML

No-Knowledge Alarms for Misaligned LLMs-as-Judges

Andr\'es Corrada-Emmanuel This is my paper
classification cs.AI stat.ML
keywords LLM-as-a-judgemisalignment detectionlogical consistencyQ-simplexunsupervised evaluationlinear feasibilityanswer-key independenceensemble monitoring
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LLMs-as-judges are used to grade other LLMs, but when no human answer key exists, supervising the judges threatens to become an infinite chain of monitors. This paper claims that the counts of how judges agree and disagree on a finite multiple-choice test are enough to detect when at least one judge is failing a user-specified accuracy requirement. The detection is a logical certificate: if, at every possible answer key, linear constraints say no judge can meet the requirement, then the ensemble necessarily contains a misaligned member. No false positives are possible, although the alarm cannot name the bad judge, and a test without any real ground truth cannot be certified by this logic. The paper demonstrates the alarm on 25 MT-Bench pair comparisons graded by two judges.

Core claim

At a fixed but unknown point in the Q-simplex—the tuple of counts of each label in the answer key—the paper derives linear label axioms relating a judge's observed response counts to its unknown per-true-label correct counts. Together with the bounds that correct counts lie between zero and the observed counts, these equations define the set of possible correct-grade counts for each judge at that point. The alarm scans all Q-simplex points and asks whether any point allows every judge to meet a user-specified accuracy threshold on every label. If no point does, the alarm fires: the observed grading pattern is logically incompatible with all judges being aligned, so at least one judge violate

What carries the argument

The Q-simplex is the integer space of possible answer-key label counts summing to the test size, representing total ignorance of the true key. The label axioms are linear equations, one per label, that express an observed response count in terms of unknown counts by true label; for example, with three labels the count of correct model-a grades equals the number of model-a items in the key minus observed non-a grades plus certain cross-label response terms. The alarm's engine is a linear feasibility check: at each simplex point, the axioms and box constraints delineate which accuracies each judge could possibly have, and if the intersection over judges is empty for every candidate answer key,

Load-bearing premise

The whole alarm rests on the test having a real correct answer key; without one, an observed inconsistency could be the test's fault or the key's fault, not the judges'.

What would settle it

For a small three-label test, enumerate every possible answer key on the Q-simplex and, for each, every assignment of true labels to observed judge grades; if any such assignment lets every judge meet the user's accuracy threshold while the alarm's feasibility check says none does, the no-false-positive claim is refuted. Running this exhaustive search on the paper's own 25-comparison count table, and on any synthetic count table, would settle whether the m=1 axioms are complete.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • An ensemble of judges can be audited for gross misalignment without any answer key: the alarm's certificate is a proof of inconsistency, not a statistical suspicion.
  • The alarm never says which judge is the bad one; the same observed counts can make different judges look misaligned at different possible answer keys.
  • The logic extends up a ladder of agreement statistics: if one-judge grade counts miss a misalignment, pair-level agreement counts can catch it.
  • Any suggested answer key can be treated as one more test taker, so including it in the counts can signal when the key, rather than the judges, is the inconsistent party.
  • A silent alarm certifies logical consistency only, never correctness, so the method can be used to screen ensembles but not to validate them.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Outside LLM grading, the same infeasibility test applies to any finite-label evaluation: human expert panels, automated graders, or mixed ensembles, as long as the response-count table is observed.
  • In practice, the alarm is best used as a triage screen: fire it on each new test batch and treat a firing as a signal to collect human labels or inspect the test, rather than as a verdict on any single judge.
  • The appendix's m=2 sketch implies a hierarchy of alarms: adding pair-level agreement counts can detect misalignments hidden by identical per-judge summary statistics, at the cost of much larger count tables.
  • Because the conclusion is conditional on the test having a valid ground truth, the alarm can convict an ensemble but never exonerate one; silence means logical consistency, not correctness.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. The paper proposes 'no-knowledge alarms' for detecting misalignment among LLM judges. The setting is a multiple-choice grading task whose true answer key is unknown. The authors model the unknown counts of true labels as a point Q in an integer simplex, and impose linear constraints on each judge's confusion matrix: row sums are the observed grade counts (Eq. 2) and column sums equal Q (Eq. 3, the 'label axioms'). The alarm is defined as follows: if, for every point Q in the simplex, it is infeasible for all judges to simultaneously satisfy a user-specified per-label accuracy threshold, then at least one judge must violate that threshold. The paper claims this occurs for a running example of 25 MT-Bench pair comparisons with two judges (gpt4 and authors), and concludes that logical consistency alone can detect misalignment with no false positives, under the assumption that the test has a ground truth.

Significance. If the central claim were correct, the paper would offer a parameter-free, logically sound certificate for detecting that an ensemble of judges cannot all meet a grading-ability requirement, without access to ground truth. The infeasibility argument itself is valid as a conditional: if no Q-simplex point permits all judges to meet the thresholds, then the actual (unknown) Q cannot either, so some judge is misaligned. This is a useful observation and the derivation of the label axiom is correct. However, the paper's only concrete demonstration is false under the stated algorithm: the running example does not trigger the alarm. The positive claim that such alarms 'can detect' misalignment is therefore unsupported as written, and the abstract overstates the scope by omitting the ground-truth assumption that the appendix itself concedes.

major comments (2)
  1. [§2.3, Eq. (2)-(3), Tables 1-3] The alarm as described cannot fire on the running example. Take Q=(5,15,5). For gpt4, the confusion matrix [[3,2,0],[2,8,0],[0,5,5]] has row sums (5,10,10), column sums Q, and diagonals (3,8,5), all above half of Q's components. For authors, [[3,1,0],[2,14,2],[0,0,3]] has row sums (4,18,3), column sums Q, and diagonals (3,14,3), also above half. Both matrices satisfy Eq. (2) and Eq. (3), so at this Q-simplex point both judges can meet the 50% requirement. The all-points scan therefore does not fire, contradicting the statement in §2.3 that 'this is the case for our running example' and Appendix A.1's claim that the m=1 axioms suffice. Adding Table 3's m=2 joint counts does not obviously repair the example: the same Q remains feasible under a joint assignment consistent with the observed agreement counts. The paper's central detection claim is unsupported as written.
  2. [Abstract and §A.4] The abstract claims detection 'with no false positives' that at least one judge is violating a grading ability requirement, without stating the assumption that the test has a valid ground truth. Section A.4 concedes that 'logical necessity cannot establish who is the wrong party here' when the test or answer key may itself be flawed. Thus an alarm firing does not by itself establish judge misalignment unless a ground-truth answer key is assumed. The abstract should carry the same qualification as Section 1, and the 'no false positives' claim should be scoped to the existence of a ground truth.
minor comments (3)
  1. [§2.2, Eq. (2)] The inequalities in Eq. (2) are printed backwards: '0≥R_{ℓi,ℓtrue}≥R_{ℓi}' should be '0≤R_{ℓi,ℓtrue}≤R_{ℓi}'. As written, the constraint is impossible for positive row counts.
  2. [Tables 1 and 2] The table captions appear to be swapped relative to the percentages quoted in the text. The figures 25%/64%/57% for the authors and 50%/86%/14% for gpt4 correspond to the other table than the one cited.
  3. [§2.1 and Table 3] The text introduces Table 3 but then says the development uses only each grader's marginal votes, leaving the role of the joint agreement counts unclear. Clarify whether Table 3 is used in the alarm computation or is only illustrative of the m=2 axioms.

Circularity Check

0 steps flagged

No circularity: the alarm is a logical consistency certificate whose inputs (grade counts, thresholds, label-sum identities) do not assume the target conclusion.

full rationale

The paper's derivation chain is self-contained and non-circular. The alarm takes observed per-judge grade marginals (e.g., gpt4 (5,10,10) and authors (4,18,3)), a user-specified accuracy requirement (at least 50% per label), and the algebraic constraints Eq. (1)-(3). Eq. (2) is just row-sum bounds on the unknown confusion matrix; Eq. (3) is a linear identity obtained by expanding R_l = sum_{l'} R_{l,l'} and Q_a = R_{a,a}+R_{b,a}+R_{c,a} and canceling terms (Section A.5). No parameter is fitted to any datum and then renamed as a prediction; the 'alarm' is the feasibility of a system of linear inequalities over the Q-simplex. The claim 'no false positives' follows by construction: if a ground-truth Q point existed with both judges meeting the threshold, then at that point the constraints would be feasible, so universal infeasibility implies at least one judge violates the threshold. This is a logical certificate, not an empirical prediction, and it does not reduce to its inputs in the sense prohibited by the circularity rules. No load-bearing self-citations appear; the only cited prior work (Platanios et al. 2014) is presented as a precursor and not used as the justification for the alarm's soundness. The paper's own limitation discussion (Section A.4) concedes that logical necessity cannot identify the wrong party when the test or answer key is suspect and cannot warn when all experts share a mistaken ground truth; that confession of scope is evidence against circularity rather than evidence for it. A separate correctness concern -- the skeptic's explicit Q=(5,15,5) confusion matrices satisfying all stated constraints, suggesting the running example's alarm may not fire as written -- is a soundness/implementation issue, not a circularity issue, and under the review rules does not raise the circularity score. Overall score 0.

Axiom & Free-Parameter Ledger

0 free parameters · 6 axioms · 0 invented entities

No free parameters are fitted: the thresholds are user-specified and the observed counts are data. The central feasibility calculation rests on the domain assumption that a ground truth exists; the paper itself flags this in Section A.4. No new physical or conceptual entities are introduced; the Q-simplex is a naming of the integer composition space of answer keys.

axioms (6)
  • domain assumption There exists a ground truth answer key assigning one of R labels to each item, corresponding to a point in the Q-simplex.
    Section 1 states the analysis proceeds 'under the assumption that they are valid and have a ground truth', and Section A.4 discusses what happens if this fails. The 'violating grading ability' claim is only meaningful relative to such a key.
  • domain assumption Observed grade counts are complete and accurate, with each grader giving exactly one of the R labels per item.
    Section 2.1 uses observed marginal counts R^{ℓ_i} as fixed inputs; any recording error or abstention would change the feasibility test.
  • standard math The label-axiom identities linking observed responses to by-true-label counts are algebraic identities, e.g., Eq. (3).
    Appendix A.5 proves the label 'a' axiom by linear cancellation from Q_a = R^{a_i,a}+R^{b_i,a}+R^{t_i,a}. Such identities are necessary conditions for any valid confusion matrix.
  • domain assumption User-specified per-label accuracy thresholds (e.g., ≥50% on every label) are the definition of aligned grading.
    The alarm only detects violations of the threshold chosen by the user; it does not justify the threshold itself, as stated in Section 3.
  • domain assumption The three grading labels (model a, model b, tie) are exhaustive and mutually exclusive for each pair comparison.
    Used in the running example; the weighted-vote procedure in Section A.3 constructs exactly one label per comparison.
  • standard math The true label counts sum to Q: sum over labels of Q_label equals Q (Eq. 1).
    Defines the Q-simplex and is used throughout Section 2.1 as the space of possible ground truths.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of No-Knowledge Alarms for Misaligned LLMs-as-Judges." pith.science (2026). https://pith.science/paper/2O5ASXQB

@misc{pith2026250908593,
  author       = {Pith},
  title        = {Pith review of: No-Knowledge Alarms for Misaligned LLMs-as-Judges},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2O5ASXQB}},
  note         = {Machine review of arXiv:2509.08593}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

If we use LLMs as judges to evaluate the complex decisions of other LLMs, who or what monitors the judges? Infinite monitoring chains are inevitable whenever we do not know the ground truth of the decisions by experts and we do not want to trust them. One way to ameliorate our evaluation uncertainty is to exploit the use of logical consistency between disagreeing experts. By observing how LLM judges agree and disagree while grading other LLMs, we can compute the only possible evaluations of their grading ability. For example, if two LLM judges disagree on which tasks a third one completed correctly, they cannot both be 100\% correct in their judgments. This logic can be formalized as a Linear Programming problem in the space of integer response counts for any finite test. We use it here to develop no-knowledge alarms for misaligned LLM judges. The alarms can detect, with no false positives, that at least one member or more of an ensemble of judges are violating a user specified grading ability requirement.

Figures

Figures reproduced from arXiv: 2509.08593 by Andr\'es Corrada-Emmanuel.

Figure 1
Figure 1. Figure 1: Possible number of correct grades by gpt4 and authors given the assumption that the answer key for the 25 MT-Bench pair comparisons contains (Qa = 8, Qb = 9, Qt = 8) labels. 3 Conclusion A running example using 25 pair comparisons from the MT-Bench dataset were used to illustrate how the counts of grades by graders can differ enough that we can logically prove one or both must be less than 50% accurate pic… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

7 extracted references · 4 linked inside Pith

  1. [1]

    Yu, Qiang Yang, and Xing Xie

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. A survey on evaluation of large language models, 2023. URL https://arxiv.org/abs/2307.03109

    Pith/arXiv arXiv 2023

  2. [2]

    Manning, Christopher Ré, Diana Acosta-Navas, Drew A

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu...

    Pith/arXiv arXiv 2023

  3. [3]

    Mitchell

    Emmanouil Antonios Platanios, Avrim Blum, and Tom M. Mitchell. Estimating accuracy from unlabeled data. In Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence (UAI), pages 655--664, Qu \'e bec City, Canada, 2014. AUAI Press

    2014

  4. [4]

    Measurement to meaning: A validity-centered framework for ai evaluation, 2025

    Olawale Salaudeen, Anka Reuel, Ahmed Ahmed, Suhana Bedi, Zachary Robertson, Sudharsan Sundar, Ben Domingue, Angelina Wang, and Sanmi Koyejo. Measurement to meaning: A validity-centered framework for ai evaluation, 2025. URL https://arxiv.org/abs/2505.10573

    Pith/arXiv arXiv 2025

  5. [5]

    Eicher-Miller, Toby Jia-Jun Liand Meng Jiang, and Ronald A

    Annalisa Szymanski, Noah Ziems, Heather A. Eicher-Miller, Toby Jia-Jun Liand Meng Jiang, and Ronald A. Metoyer. Limitations of the LLM -as-a- Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks Proceedings of the 30th International Conference on Intelligent User Interfaces , 2025. URL https://dl.acm.org/doi/10.1145/3708359.3712091. Archive...

    arXiv 2025

  6. [6]

    Provably safe systems: the only path to controllable agi, 2023

    Max Tegmark and Steve Omohundro. Provably safe systems: the only path to controllable agi, 2023. URL https://arxiv.org/abs/2309.01933

    Pith/arXiv arXiv 2023

  7. [7]

    Principal–agent problem --- Wikipedia , the free encyclopedia, 2025

    Wikipedia contributors . Principal–agent problem --- Wikipedia , the free encyclopedia, 2025. URL https://en.wikipedia.org/w/index.php?title=Principal [Online; accessed 3-September-2025]

    2025

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.