Pith Number

pith:UFNMQTM3

pith:2026:UFNMQTM3CZRSICCJNKAEVPYBPU

not attested not anchored not stored refs pending

Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines

Sadman Kabir Soumik

Style bias dominates LLM judges at 0.76-0.92 strength across models, dwarfing position bias, while combined debiasing improves agreement for some judges by over 11 points.

arxiv:2604.23178 v2 · 2026-04-25 · cs.AI

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{UFNMQTM3CZRSICCJNKAEVPYBPU}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

Style bias is the dominant bias (0.76-0.92 across all models), far exceeding position bias (<= 0.04), yet has received minimal research attention.

C2weakest assumption

The chosen benchmarks (MT-Bench, LLMBar, and the custom set) and bias measurement protocols accurately isolate the targeted biases without introducing confounding artifacts from the test data itself or from how responses were generated.

C3one line summary

Style bias dominates LLM-as-a-Judge systems far more than position bias, with debiasing strategies providing model-dependent gains and public tools released for replication.

Cited by

1 paper in Pith

Does Capability Transfer to Subjective Behavior -- and Would Our Instruments Tell Us? A Self-Evolving, Trust-by-Construction Evaluation Paradigm

Receipt and verification

First computed	2026-06-25T01:18:38.181048Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

a15ac84d9b16632408496a804abf017d37fff889dd71edf862751ed24cb6be94

Aliases

arxiv: 2604.23178 · arxiv_version: 2604.23178v2 · doi: 10.48550/arxiv.2604.23178 · pith_short_12: UFNMQTM3CZRS · pith_short_16: UFNMQTM3CZRSICCJ · pith_short_8: UFNMQTM3

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/UFNMQTM3CZRSICCJNKAEVPYBPU \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: a15ac84d9b16632408496a804abf017d37fff889dd71edf862751ed24cb6be94

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "cfc3a751c58c0db75e5635e0c86944b4554226db4975fafa682d72901497c7d7",
    "cross_cats_sorted": [],
    "license": "http://creativecommons.org/licenses/by/4.0/",
    "primary_cat": "cs.AI",
    "submitted_at": "2026-04-25T07:18:30Z",
    "title_canon_sha256": "33773dbbee0db37c531b7fba373163b94d36dcd257911d480bd5ae8c2699e8e5"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2604.23178",
    "kind": "arxiv",
    "version": 2
  }
}