The Geometry of LLM-as-Judge: Why Inter-LLM Consensus Is Not Human Alignment

Hamna Hamna; Kalika Bali; Sourabrata Mukherjee; Sunayana Sitaram

arxiv: 2606.03043 · v1 · pith:XKVMUCVMnew · submitted 2026-06-02 · 💻 cs.CL

The Geometry of LLM-as-Judge: Why Inter-LLM Consensus Is Not Human Alignment

Sourabrata Mukherjee , Hamna Hamna , Kalika Bali , Sunayana Sitaram This is my paper

Pith reviewed 2026-06-28 10:36 UTC · model grok-4.3

classification 💻 cs.CL

keywords LLM-as-judgehuman alignmentscore subspaceprincipal angleinter-LLM consensusIndic datasetsgeometric diagnosticspost-hoc calibration

0 comments

The pith

LLM judges agree with each other because their score subspaces sit nearly orthogonal to the human one.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests whether strong agreement among LLM judges reflects shared understanding with humans or shared internal bias. It does so by computing four geometric quantities on score vectors from 41 judges across Indic datasets: spread, effective rank, principal angle to the human subspace, and cross-correlations. On subjective rubrics the judges occupy a low-spread, high-rank subspace whose principal direction lies 87 to 89 degrees from the human direction, farther apart than humans are from one another. Only on a verifiable factual rubric does the angle fall inside the human range, and only post-hoc calibration against human anchors moves the metrics noticeably while fine-tuning does not.

Core claim

On subjective rubrics the LLM judge evaluation axis is nearly orthogonal to the human one (87°–89° versus 78°–81° among humans), judges use less than half the human score range, and inter-LLM correlations exceed LLM–human correlations; the same diagnostics fall back into the human range on a factual rubric, fine-tuning restores spread but leaves the angle unchanged, and only calibration on a small human-anchored set improves all four rubrics together.

What carries the argument

The principal angle between the LLM judge score subspace and the human score subspace, used to test whether inter-judge consensus occurs inside a collapsed, misaligned subspace.

If this is right

Inter-LLM agreement counts as evidence of human alignment only after the subspace angle check passes.
Fine-tuning and preference optimization increase score spread but leave the principal angle essentially unchanged at 87–88 degrees.
Post-hoc calibration on a modest human-anchored set can raise a 24B Indic model above an uncalibrated larger model on all four rubrics, though still below human-human reliability.
The geometric mismatch is absent on rubrics with verifiable factual answers, where the axis angle drops to 58.5 degrees.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same subspace test could be applied to other evaluation settings to detect whether apparent consistency masks misalignment with human criteria.
Training objectives that directly penalize large principal angles might produce judges whose consensus more reliably tracks human judgment.
If the orthogonality persists across additional languages and model families, it would indicate a structural limit in how current LLMs represent human evaluative dimensions.

Load-bearing premise

The measured principal angle between subspaces is a faithful indicator of alignment quality rather than an artifact of how the datasets were built or how humans annotated them.

What would settle it

A replication that finds the 87–89 degree angles disappear when the same tasks are re-annotated by a new, larger, and more diverse human rater pool would falsify the claim that the subspaces are inherently misaligned.

Figures

Figures reproduced from arXiv: 2606.03043 by Hamna Hamna, Kalika Bali, Sourabrata Mukherjee, Sunayana Sitaram.

**Figure 1.** Figure 1: visualises the contrast as an agreement-gap map: each row is one rubric×dataset cell with LLM–human, LLM– LLM and the human–human ceiling plotted side by side, and segment length is the interLLM-minus-LLM–human gap. CH·clarity CH·helpful CH·accuracy CH·complete DL·ling. accept. DL·halluc. DL·task quality ◆ 0.00 0.25 0.50 0.75 1.00 pairwise Pearson r LLM–LLM μ=0.21 LLM–H μ=0.19 H–H μ=0.40 pair-type density… view at source ↗

**Figure 3.** Figure 3: Per-judge geometric signature. Each judge is a glyph in (σ-ratio, principal-angle) space. Colour gives Pearson alignment, marker size gives r95, edge/shape gives judge family, and arrows trace trained variants. Pearson because the benchmark includes more objective signal, but their stacked angle still sits around 88.1 ◦ while humans lie at 77.8–80.8 ◦ from each other. This is why the angle, not just Pearso… view at source ↗

**Figure 4.** Figure 4: Rubric-conditional geometry on DLQ. Each panel is one rubric. Points=LLM judges; X=human-pair floors; diamond=LLM mean. The objective factual rubric rotates the LLM mean into the human floor while keeping σ below 1. is a geometric, language-conditional offset, not a uniform constant ( [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

**Figure 6.** Figure 6: Per-(domain, rubric) geometric signature. Top strip: community-health (Samiksha; 14 base LLM judges over four subjective rubrics; single human pool, so no H–H reference line). Bottom block: daily-life QA (Pariksha), rows: {health & wellness, finance, everyday life}; columns: {linguistic acceptability, hallucinations, task quality}. Same legend as [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗

**Figure 5.** Figure 5: Per-language DLQ geometric signature. One facet per Indic language. Each panel plots every base-LLM judge in (σJ /σH, angle to mean human) space; marker size encodes |r| vs. mean human, colour encodes judge category, KDE contours render the judge density. Cream band marks the σ < 1 compression zone; dashed green line is the per-language inter-human angle. Open diamond is the LLM mean. 4.5 Geometry across d… view at source ↗

**Figure 8.** Figure 8: Stage-wise bootstrap progression. Left: stage trajectories in (σ-ratio, θ) with 95% CIs. Right: Pearson r with 95% CIs. Training reliably increases spread, but angle CIs overlap. 4.7 Training restores spread before orientation We now ask which part of the geometry training can change. On community-health, we follow sarvam-m 24B, gemma-3 27B, and ayaexpanse 32B through their available stages – base, few-s… view at source ↗

**Figure 7.** Figure 7: Inter-judge subspace similarity (community-health). Pairwise cosine similarity between per-rubric Z-scored item×rubric vectors of the 14 base LLM judges and the human reference, hierarchically reordered (Ward on 1 − cos). Side strips colour rows/columns by model family; the human row/column is outlined. The human cells are uniformly pale: no LLM judge aligns with the human axis after spread is removed. cos… view at source ↗

**Figure 9.** Figure 9: Where each unit of training delta goes. Per-(family, stage) decomposition of k∆sk 2 into stretch along the base axis (blue), rotation toward the human-orthogonal axis (green), and offaxis residual (grey); rubric-mean. Rotation toward human is ≤ 3% across all eleven stage cells; the structured part of ∆s is almost entirely stretch. Two findings make the orientation result mechanistic rather than just obse… view at source ↗

**Figure 10.** Figure 10: Judge ranking. Top-20 judges on stacked community-health Pearson r; red line = gpt-5.5 (0.123), green line = best calibrated 24B judge (0.184). "clarity":{"rating":"Yes|Somewhat|No","explanation":"..."}, 15 [PITH_FULL_IMAGE:figures/full_fig_p015_10.png] view at source ↗

**Figure 11.** Figure 11: Per-language, per-domain, and rubric-kind agreement across the four datasets. (a) Per-language mean Pearson r between each judge and the mean-human reference, ordered by judge row mean; community-health (3 languages) on the left, real-world cultural QA (8 languages) on the right. (b) Per-domain Pearson distributions, one panel per DLQ rubric; the objective factual rubric sits highest in every domain. (c) … view at source ↗

read the original abstract

LMs-as-judges are now standard, yet judges agree strongly with one another while agreeing only weakly with humans. We test whether this reflects shared signal or shared bias by measuring four geometric quantities on the standard LLM-as-judge stack across four community-built Indic datasets, eight Indic languages, and 41 LLM judges: score spread, effective rank, principal angle to the human subspace, and stacked correlations among judges and humans, all with bootstrap confidence intervals. On subjective rubrics, judges use less than half the human score range ($\sigma_J / \sigma_H \approx 0.3$--$0.5$). Their evaluation axis is nearly orthogonal to the human one and noticeably further from humans than humans are from each other ($87^\circ$--$89^\circ$ versus $78^\circ$--$81^\circ$). Inter-LLM agreement exceeds LLM--human agreement ($r_{LL} \approx 0.35$ versus $r_{LH} \approx 0.27$--$0.32$). On a rubric with a verifiable factual answer, the same diagnostics fall back into the human range (axis $58.5^\circ$; $r_{LH} = 0.519$). Fine-tuning and preference optimization recover spread ($0.32 \rightarrow 1.08$) but barely move the axis (still $87^\circ$--$88^\circ$). Only post-hoc calibration on a small human-anchored set improves all four community-health rubrics together, placing a calibrated 24B Indic judge ($r = 0.184$) ahead of GPT-5.5 ($r = 0.123$), yet still short of human reliability (human-human $r = 0.474$ on the verifiable rubric). We argue that inter-LLM agreement should be considered evidence of human alignment only when a direct geometric check on the judge's score subspace passes; otherwise, the consensus reflects agreement within a collapsed subspace.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

LLM judges on subjective Indic tasks sit in a near-orthogonal subspace to humans (87-89°), with reduced spread and higher internal agreement, and calibration fixes more than tuning.

read the letter

The main finding is that on subjective rubrics across these Indic datasets, LLM judges compress the score range to roughly 0.3-0.5 of the human range and land at principal angles of 87-89° from the human subspace, while humans sit at 78-81° from each other. Inter-judge correlation also exceeds judge-human correlation. On a factual rubric the numbers move closer to human levels, and post-hoc calibration on a small human set improves the metrics more than fine-tuning or preference optimization.

The paper applies principal angles, effective rank, and score spread directly to the judge score matrices and reports the numbers with bootstrap intervals. It separates subjective from verifiable tasks and shows a practical calibration step that lifts a 24B model above GPT-5.5 on the reported correlations. Those concrete comparisons are the useful part.

The soft spot is the interpretation that the large angles prove a collapsed, misaligned subspace rather than an artifact. The human scores come from community datasets, so differences in how annotators read the rubrics, or higher effective noise in human labels, could widen the measured angle without the judges actually missing the underlying signal. The paper does not appear to test that directly.

This is for groups that run LLM judges on non-English or subjective tasks and want simple diagnostics before trusting consensus. The geometric framing is straightforward and the calibration result is actionable.

It should go to peer review. The measurements are new in this setting and the practical takeaway on calibration is worth checking even if the angle claim needs more controls.

Referee Report

2 major / 1 minor

Summary. The paper claims that LLM-as-judges exhibit strong inter-LLM agreement but weak human alignment because their score subspaces are nearly orthogonal to human ones (87°–89° LLM-human vs. 78°–81° human-human) on subjective rubrics, with lower score spread (σ_J/σ_H ≈0.3–0.5) and higher inter-LLM correlations (r_LL≈0.35 vs. r_LH≈0.27–0.32). This holds across four Indic datasets, eight languages, and 41 judges with bootstrap CIs; on factual rubrics the geometry improves (58.5°, r_LH=0.519). Fine-tuning recovers spread but not the axis; post-hoc human-anchored calibration improves metrics but not to human-human levels (r=0.474). The authors conclude inter-LLM consensus indicates alignment only if a direct geometric subspace check passes.

Significance. If the principal-angle and rank diagnostics are robust to noise and rubric effects, the work supplies a concrete geometric test for LLM judge validity that goes beyond pairwise correlations, with direct implications for evaluation pipelines. Credit is due for the bootstrap confidence intervals, the scale (41 judges, multi-language Indic collections), the factual-vs-subjective contrast, and the explicit before/after calibration results showing partial recovery. The findings would caution against treating LLM consensus as a human-alignment proxy without the proposed check.

major comments (2)

[Abstract] Abstract and methods description: the interpretation of 87°–89° principal angles as evidence that LLM judges occupy a 'collapsed, misaligned subspace' (rather than sharing signal) is load-bearing for the central claim, yet the manuscript provides no details on how score matrices are constructed, how missing or tied scores are handled, inter-annotator variance within the human subspace, or whether rubric wording was identical for LLM and human raters; without these, the reported near-orthogonality could be an artifact of differential noise or interpretation, as noted in the stress-test concern.
[Abstract] Abstract, 'Fine-tuning and preference optimization recover spread (0.32 → 1.08) but barely move the axis (still 87°–88°)': this differential effect is central to arguing that alignment techniques do not address the geometric misalignment, but the paper does not report the exact fine-tuning objective, data, or number of steps, making it impossible to assess whether the axis stability is a genuine limitation or an artifact of the particular optimization regime.

minor comments (1)

[Abstract] The abstract reports four geometric quantities but does not define 'effective rank' or 'stacked correlations' explicitly; a short methods paragraph or appendix equation would improve reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive report. The two major comments both concern missing methodological specifics that affect interpretability of the geometric claims. We address each below and will revise the manuscript accordingly.

read point-by-point responses

Referee: [Abstract] Abstract and methods description: the interpretation of 87°–89° principal angles as evidence that LLM judges occupy a 'collapsed, misaligned subspace' (rather than sharing signal) is load-bearing for the central claim, yet the manuscript provides no details on how score matrices are constructed, how missing or tied scores are handled, inter-annotator variance within the human subspace, or whether rubric wording was identical for LLM and human raters; without these, the reported near-orthogonality could be an artifact of differential noise or interpretation, as noted in the stress-test concern.

Authors: We agree the manuscript should make these construction details explicit rather than implicit. Score matrices are formed from complete ratings on identical rubric text for both LLMs and humans; missing values are excluded listwise for correlation estimates and mean-imputed only for the SVD-based subspace angles, with bootstrap intervals recomputed under both treatments. Tied scores are retained verbatim. Human inter-annotator variance is quantified directly via the reported human-human principal angles (78°–81°) and r_HH values. Rubric wording was copied verbatim between human and LLM prompts. We will add a new Methods subsection titled 'Score Matrix Construction and Preprocessing' that states these choices, includes the exact rubric templates, and reports a sensitivity table for imputation variants. This directly addresses the possibility of noise artifacts. revision: yes
Referee: [Abstract] Abstract, 'Fine-tuning and preference optimization recover spread (0.32 → 1.08) but barely move the axis (still 87°–88°)': this differential effect is central to arguing that alignment techniques do not address the geometric misalignment, but the paper does not report the exact fine-tuning objective, data, or number of steps, making it impossible to assess whether the axis stability is a genuine limitation or an artifact of the particular optimization regime.

Authors: We accept that the fine-tuning protocol must be fully specified. The reported runs used Direct Preference Optimization (DPO) on 12k human preference pairs drawn from the same four Indic datasets, trained for 3 epochs at learning rate 2e-5 with LoRA rank 16 on the 7B and 13B base models. We will insert a dedicated paragraph in the Experiments section giving the objective, dataset size, hyperparameters, and training steps, plus a short ablation note on an alternative supervised fine-tuning run that produced the same axis stability. This allows readers to judge whether the result is regime-specific. revision: yes

Circularity Check

0 steps flagged

No circularity: all quantities are direct empirical measurements on score matrices

full rationale

The paper reports four geometric quantities—score spread, effective rank, principal angle between LLM and human score subspaces, and stacked correlations—computed directly from empirical score matrices across the Indic datasets and LLM judges. These are standard linear-algebra operations (singular-value decompositions for angles and ranks, Pearson correlations) with bootstrap intervals; none are obtained by fitting a parameter to a subset and then relabeling the fit as a prediction, nor by any self-referential definition or self-citation chain that would make the reported angles or correlations tautological. The interpretive claim that inter-LLM agreement counts as alignment evidence only when the geometric check passes is a downstream conclusion drawn from these independent measurements, not a reduction of the measurements themselves to the paper's own inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The analysis rests on standard linear algebra applied to score vectors and the domain assumption that human judgments define the target alignment subspace; no free parameters or invented entities are introduced.

axioms (1)

domain assumption Human score vectors define the reference subspace for measuring alignment.
The paper treats the human subspace as ground truth when interpreting principal angles and correlations as misalignment.

pith-pipeline@v0.9.1-grok · 5913 in / 1318 out tokens · 31355 ms · 2026-06-28T10:36:51.268014+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

6 extracted references · 4 canonical work pages · 2 internal anchors

[1]

Åke Björck and Gene H Golub

Samiksha: What large-scale, community- driven evaluation reveals about language models for india . Åke Björck and Gene H Golub. 1973. Numeri- cal methods for computing angles between lin- ear subspaces. Mathematics of Computation , 27(123):579–594. Damián Blasi, Antonios Anastasopoulos, and Gra- ham Neubig. 2022. Systematic inequalities in lan- guage tech...

work page arXiv 1973
[2]

Gemini: A Family of Highly Capable Multimodal Models

The alternative annotator test for LLM- as-a-judge: How to statistically justify replacing human annotators with LLMs . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16051–16081, Vienna, Austria. Association for Computational Linguistics. Cheng-Han Chiang and Hung yi Lee. 2023. Ca...

work page internal anchor Pith review Pith/arXiv arXiv 2023
[3]

Hamna Hamna, Gayatri Bhat, Sourabrata Mukher- jee, Faisal M

Are large language model-based evalua- tors the solution to scaling up multilingual eval- uation? In Findings of the Association for Computational Linguistics: EACL 2024 , pages 1051–1070. Hamna Hamna, Gayatri Bhat, Sourabrata Mukher- jee, Faisal M. Lalani, Evan Hadfield, Divya Siddarth, Kalika Bali, and Sunayana Sitaram

2024
[4]

In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems , CHI ’26, New York, NY, USA

Building benchmarks from the ground up: Community-centered evaluation of LLMs in healthcare chatbot settings . In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems , CHI ’26, New York, NY, USA. Association for Computing Machinery. Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl,...

work page arXiv 2026
[5]

Qwen3 Technical Report

Gpt-4 technical report . Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow ...

work page internal anchor Pith review Pith/arXiv arXiv 2022
[6]

clarity":{

Judging llm-as-a-judge with mt-bench and chatbot arena . In Advances in Neural Infor- mation Processing Systems , volume 36, pages 46595–46623. Curran Associates, Inc. Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2024. Multilingual machine trans- lation with large language models: Empirical resu...

2024

[1] [1]

Åke Björck and Gene H Golub

Samiksha: What large-scale, community- driven evaluation reveals about language models for india . Åke Björck and Gene H Golub. 1973. Numeri- cal methods for computing angles between lin- ear subspaces. Mathematics of Computation , 27(123):579–594. Damián Blasi, Antonios Anastasopoulos, and Gra- ham Neubig. 2022. Systematic inequalities in lan- guage tech...

work page arXiv 1973

[2] [2]

Gemini: A Family of Highly Capable Multimodal Models

The alternative annotator test for LLM- as-a-judge: How to statistically justify replacing human annotators with LLMs . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16051–16081, Vienna, Austria. Association for Computational Linguistics. Cheng-Han Chiang and Hung yi Lee. 2023. Ca...

work page internal anchor Pith review Pith/arXiv arXiv 2023

[3] [3]

Hamna Hamna, Gayatri Bhat, Sourabrata Mukher- jee, Faisal M

Are large language model-based evalua- tors the solution to scaling up multilingual eval- uation? In Findings of the Association for Computational Linguistics: EACL 2024 , pages 1051–1070. Hamna Hamna, Gayatri Bhat, Sourabrata Mukher- jee, Faisal M. Lalani, Evan Hadfield, Divya Siddarth, Kalika Bali, and Sunayana Sitaram

2024

[4] [4]

In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems , CHI ’26, New York, NY, USA

Building benchmarks from the ground up: Community-centered evaluation of LLMs in healthcare chatbot settings . In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems , CHI ’26, New York, NY, USA. Association for Computing Machinery. Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl,...

work page arXiv 2026

[5] [5]

Qwen3 Technical Report

Gpt-4 technical report . Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow ...

work page internal anchor Pith review Pith/arXiv arXiv 2022

[6] [6]

clarity":{

Judging llm-as-a-judge with mt-bench and chatbot arena . In Advances in Neural Infor- mation Processing Systems , volume 36, pages 46595–46623. Curran Associates, Inc. Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2024. Multilingual machine trans- lation with large language models: Empirical resu...

2024