Pith. sign in

REVIEW 1 major objections 1 minor 69 cited by

Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge

T0 review · 1 major / 1 minor · reviewed 2026-05-15 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLM-as-a-Judge systems carry 12 measurable biases that automated tests can isolate and that persist in specific tasks.

desk verdict The paper lists 12 biases in LLM judges and offers CALM as an automated quantifier, but the method's ability to isolate each bias cleanly is not yet demonstrated. read the letter →

arxiv 2410.02736 v2 pith:VO6XP5ZK submitted 2024-10-03 cs.CL cs.AI

Jiayi Ye , Yanbo Wang , Yue Huang , Dongping Chen , Qihui Zhang , Nuno Moniz , Tian Gao , Werner Geyer
show 4 more authors
Chao Huang Pin-Yu Chen Nitesh V Chawla Xiangliang Zhang
This is my paper · ORCID
classification cs.CLcs.AI
keywords LLM-as-a-JudgebiasquantificationevaluationreliabilityautomatedframeworklanguagemodelsprejudicedetectionCALM
checked against Cost.FunctionalEquation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to establish that LLM-as-a-Judge, already used for benchmarks and training rewards, is undermined by 12 distinct biases that reduce its reliability. It introduces the CALM framework, which applies automated principle-guided modifications to inputs in order to quantify each bias separately across popular language models. Experiments show that while overall performance is strong, certain tasks still display significant biases, implying that the method requires further refinement before it can be trusted without reservation. A sympathetic reader would care because biased judges can distort evaluation scores and training signals throughout AI development pipelines.

What carries the argument

The CALM framework, which isolates and measures each of the 12 biases by applying automated principle-guided modifications to evaluation inputs.

What would settle it

Repeating the CALM measurements on the same model and inputs but obtaining substantially different bias scores when a different set of guiding principles is used would indicate that the isolation procedure does not reliably separate the biases.

Watch

Extended reading notes

Core claim

The paper claims that its CALM framework systematically quantifies 12 potential biases in LLM-as-a-Judge through automated and principle-guided input modifications, with empirical results across multiple models indicating that significant biases persist in certain specific tasks even when overall performance remains commendable.

Load-bearing premise

That automated principle-guided modifications can cleanly isolate each bias without introducing new confounding effects or missing interactions between biases.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 1 minor

Summary. The paper identifies 12 key biases in LLM-as-a-Judge, proposes the CALM automated bias quantification framework that uses principle-guided modifications to measure each bias, reports experiments across multiple popular LLMs showing that advanced models exhibit commendable overall performance yet retain significant biases on specific tasks, and concludes there remains room for improvement in reliability while offering suggestions for cautious application.

Significance. If the modifications in CALM can be shown to isolate individual biases without confounding, the work would be significant for the many benchmarks and training pipelines that rely on LLM judges, by supplying a systematic diagnostic that could guide mitigation and increase trust in automated evaluation.

major comments (1)
  1. [CALM framework and experimental results] The central claim that CALM quantifies 12 distinct biases rests on the assumption that each automated principle-guided modification affects only its target bias dimension. No ablation on modification prompts, cross-bias correlation analysis, or human validation of isolated effects is described, so interactions or model-induced artifacts cannot be ruled out; this directly undermines interpretability of the per-bias scores and the headline conclusion that biases persist in specific tasks.
minor comments (1)
  1. [Abstract and Experiments] The abstract states results cover 'multiple popular language models' but provides no model names, sizes, or prompting details; these should be listed explicitly in the experimental setup section.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive comment on the isolation of biases within the CALM framework. We respond to the major comment below.

read point-by-point responses

  1. Referee: The central claim that CALM quantifies 12 distinct biases rests on the assumption that each automated principle-guided modification affects only its target bias dimension. No ablation on modification prompts, cross-bias correlation analysis, or human validation of isolated effects is described, so interactions or model-induced artifacts cannot be ruled out; this directly undermines interpretability of the per-bias scores and the headline conclusion that biases persist in specific tasks.

    Authors: We agree that empirical confirmation of isolated effects is essential for the interpretability of the per-bias scores. Each modification in CALM is constructed from explicit, bias-specific principles that alter only the targeted dimension (e.g., swapping option order for positional bias while holding content fixed). This principle-guided design aims to minimize confounding by construction. Nevertheless, the current manuscript does not include ablations on the modification prompts, cross-bias correlation matrices, or human validation of the isolated effects. To address this directly, we will add (i) an ablation study varying prompt phrasing for a subset of biases, (ii) pairwise correlation analysis across all 12 bias scores, and (iii) a small-scale human study verifying that the modifications produce the intended isolated changes. These additions will appear in the revised manuscript and appendix. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity detected

full rationale

The paper is an empirical study proposing the CALM framework to quantify 12 biases in LLM-as-a-Judge via automated principle-guided modifications. No equations, derivations, or self-citations appear in the provided abstract or context that reduce any central claim to its own inputs by construction. The methodology and results are presented as independent experimental outputs across multiple models, with no fitted parameters renamed as predictions or uniqueness theorems imported from prior self-work. This matches the default expectation for non-circular empirical papers; the reader's score of 2.0 and skeptic concerns address potential confounding in bias isolation (a validity issue) rather than any load-bearing step that collapses to self-definition or self-citation.

Assumptions & free parameters 0 free parameters · 1 assumptions · 1 invented entities

The central claim rests on the domain assumption that biases can be isolated through automated modifications and that the 12 listed biases are the key ones to measure.

assumptions (1)
  • domain assumption Biases in LLM-as-a-Judge can be isolated and quantified via automated principle-guided input modifications
    This is the core premise of the CALM framework described in the abstract.
invented entities (1)
  • CALM framework
    purpose: Automated bias quantification for LLM-as-a-Judge
    New method introduced by the paper; no independent evidence provided in abstract.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge." pith.science (2026). https://pith.science/paper/VO6XP5ZK

@misc{pith2026241002736,
  author       = {Pith},
  title        = {Pith review of: Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VO6XP5ZK}},
  note         = {Machine review of arXiv:2410.02736}
}
read the original abstract

LLM-as-a-Judge has been widely utilized as an evaluation method in various benchmarks and served as supervised rewards in model training. However, despite their excellence in many domains, potential issues are under-explored, undermining their reliability and the scope of their utility. Therefore, we identify 12 key potential biases and propose a new automated bias quantification framework-CALM-which systematically quantifies and analyzes each type of bias in LLM-as-a-Judge by using automated and principle-guided modification. Our experiments cover multiple popular language models, and the results indicate that while advanced models have achieved commendable overall performance, significant biases persist in certain specific tasks. Empirical results suggest that there remains room for improvement in the reliability of LLM-as-a-Judge. Moreover, we also discuss the explicit and implicit influence of these biases and give some suggestions for the reliable application of LLM-as-a-Judge. Our work highlights the need for stakeholders to address these issues and remind users to exercise caution in LLM-as-a-Judge applications.

Discussion (0). Continue with ORCID to comment.

Forward citations

Showing 60 of 69 Pith papers that cite this

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. See all 69 Pith citations

  1. CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks

    cs.CL 2026-06 conditional novelty 7.0 of 10

    CoEval generates task-specific benchmarks by rotating models through teacher, student, and judge roles, then weights questions by discriminative power and judges by panel consensus to recover accurate model rankings w...

  2. Matter to Mechanism: A Benchmark for AI Co-Scientists in Materials and Battery Research

    cs.CE 2026-06 unverdicted novelty 7.0 of 10

    Introduces the Matter to Mechanism benchmark of 2,645 structured instances and a composite metric suite for evaluating AI co-scientists on problem-to-hypothesis reasoning in battery materials research.

  3. Does Capability Transfer to Subjective Behavior -- and Would Our Instruments Tell Us? A Self-Evolving, Trust-by-Construction Evaluation Paradigm

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    Self-evolving rubric with anti-gaming fitness reveals that objective capability scaling fails to transfer to subjective LLM behaviors, with advice-restraint as the universal lowest dimension that can regress.

  4. GRASP: Deterministic argument ranking in interaction graphs

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    GRASP aggregates stable local LLM interaction judgments into global argument rankings via a convergent attack-defense propagation operator on interaction graphs, yielding higher reproducibility than holistic judging a...

  5. Recall Isn't Enough: Bounding Commitments in Personalized Language Systems

    cs.AI 2026-05 unverdicted novelty 7.0 of 10

    CBEA with LCV bounds evidence sets and validates commitments before response generation, achieving zero failures in scoped tests at 0.49-0.60 availability versus near-zero for baselines.

  6. Prompt Framing Distorts Count Based Evaluation of LLM Error Detection: Evidence from Numeric Anchoring

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    Supplying the true error count in a prompt raises LLM Count-F1 by ~0.22 but edit-level F0.5 by only ~0.04, so count agreement mostly measures prompt compliance.

  7. IatroBench: Pre-Registered Evidence of Iatrogenic Harm from AI Safety Measures

    cs.AI 2026-04 unverdicted novelty 7.0 of 10

    Frontier models systematically withhold clinically necessary guidance from laypeople that they provide to physicians on identical facts, and LLM judges fail to detect that omission harm.

  8. Safety Under Scaffolding: How Evaluation Conditions Shape Measured Safety

    cs.SE 2026-03 conditional novelty 7.0 of 10

    Map-reduce scaffolding degrades measured safety mainly by stripping multiple-choice options (40–89% of the loss is format conversion); scaffold architecture explains only 0.4% of variance and composite safety scores h...

  9. REALEDIT: Reddit Edits As a Large-scale Empirical Dataset for Image Transformations

    cs.CV 2025-02 conditional novelty 7.0 of 10

    REALEDIT provides a large-scale, real-world image editing dataset from Reddit and demonstrates that training on it improves performance on authentic user requests.

  10. Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Models

    cs.CL 2024-12 conditional novelty 7.0 of 10

    Jailbreak attacks push LLM activations outside a safety boundary, mostly in low and middle layers, and a tanh-based penalty that pulls activations back inside this boundary blocks most tested attacks with under 2% uti...

  11. JuStRank: Benchmarking LLM Judges for System Ranking

    cs.CL 2024-12 conditional novelty 7.0 of 10

    JuStRank ranks AI judges by how well their aggregated scores reproduce the Chatbot Arena human system ranking, revealing that judge realization and bias, not just model size, determine ranking quality.

  12. TREK: A Travel Reasoning and Evaluation Kit for LLM Agents in Complex Trip Planning

    cs.CL 2026-07 conditional novelty 6.5 of 10

    On 800 jointly constrained trip tasks with a deterministic scorer and achievable gold, the best of 15 LLM agents fully solves only 46.2% of feasible plans, with unstated persona needs as the universal bottleneck.

  13. Eval-Pair Matrix: Answer-Paired Meta-Evaluation of LLM Judges for Grounded RAG

    cs.CL 2026-07 conditional novelty 6.5 of 10

    Answer-paired analysis of a 3×3 GPT/Grok/Gemini judge matrix finds near-zero same-model recall bias for induced RAG grounding errors; remaining flag gaps reflect label-task mismatch.

  14. A Scalable Approach to Evaluating Moral Sensitivity in LLMs

    cs.CY 2026-07 conditional novelty 6.5 of 10

    Under morally irrelevant noise, eight LLMs preserve the semantic content of identified moral features above calibrated floors, despite significant changes in feature counts.

  15. Order Matters: LVLMs as Judges for Temporal Reasoning in Image Sequences

    cs.CV 2026-08 conditional novelty 6.0 of 10

    LVLM judges are near chance when asked to pick the correctly ordered version of an image sequence, and this temporal blindness persists after fine-tuning and at larger scale.

  16. The Authority Expectancy Effect in Multi-User Conflict

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Social authority signals can reverse or restructure LLM prioritization relative to a model's own severity baseline, an effect the paper terms the Authority Expectancy Effect.

  17. Cross-Lingual Bias in Large Language Models: A Comparative Analysis of English and Swahili

    cs.CL 2026-08 conditional novelty 6.0 of 10

    Bias in GPT-5.2 and Gemini 2.5 Flash changes rather than transfers between English and Swahili, with GPT-5.2 refusal behavior appearing only in English.

  18. LLM-as-a-Judge for Evaluating System Responses in Conversational Music Recommendation

    cs.IR 2026-07 conditional novelty 6.0 of 10

    LLM judges agree moderately with human experts when scoring conversational music recommendation responses, outperform reference-based metrics, but are not reliable enough to replace human evaluation.

  19. Grokipedia vs Wikipedia: An LLM-Based Audit of Political Neutrality along Ideologies

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Across 1,394 paired biographies, four LLM judges rated AI-written Grokipedia as less neutral than Wikipedia, with Grokipedia favoring economically right-wing politicians and Wikipedia favoring socially liberal ones.

  20. Different Teachers, Different Capabilities: Sub-1B On-Device Distillation for Structured Text Enrichment

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Distilling an 8B reasoning teacher into a 0.6B student recovers most summary quality at ~50× speed, but teacher type—not scale alone—determines which capabilities transfer.

  21. Towards Spec Learning: Inference-Time Alignment from Preference Pairs

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    Spec learning compiles brief instructions and preference pairs into readable natural-language specifications that condition LLMs at inference time and can outperform DPO on domains with dense preference signals.

  22. SOMA-SQL: Resolving Multi-Source Ambiguity in NL-to-SQL via Synthetic Log and Execution Probing

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    SOMA-SQL resolves multi-source ambiguity in NL-to-SQL using synthetic query logs and ambiguity-driven execution probing, reporting 13% average execution accuracy gains over baselines on six benchmarks.

  23. Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents

    cs.CL 2026-06 accept novelty 6.0 of 10

    Empirical study of a production multi-turn ordering agent finds LLM-as-judge recall below 25% for human-confirmed defects, missing cross-turn state issues due to limited rubric and routing.

  24. AMEL: Accumulated Message Effects on LLM Judgments

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    LLMs exhibit an accumulated message effect where conversation history saturated with positive or negative evaluations biases subsequent judgments, with larger shifts on uncertain items, a negativity asymmetry, and no ...

  25. Chain-of-Models: Cross-Model Auditing for Bias-Robust LLM Judges

    cs.CL 2026-05 conditional novelty 6.0 of 10

    Per-bias selection of a cross-family LLM auditor lifts biased-judgment accuracy from 0.805/0.824 baselines to 0.884.

  26. Auditing Multimodal LLM Raters: Central Tendency Bias in Clinical Ordinal Scoring

    cs.CV 2026-05 conditional novelty 6.0 of 10

    Multimodal LLMs exhibit central tendency bias when scoring ordinal clinical images, over-predicting low scores and under-predicting high scores even after prompt ablations.

  27. Evaluating AI-Generated Images of Cultural Artifacts with Community-Informed Rubrics

    cs.CY 2026-04 unverdicted novelty 6.0 of 10

    Community members from the UK blind community, Kerala, and Tamil Nadu helped define what counts as culturally appropriate depictions of artifacts, and the authors tested whether those definitions can be turned into re...

  28. Who Endorsed It? Measuring Authority Bias Across Expertise Levels in Language Models

    cs.CL 2026-01 reject novelty 6.0 of 10

    Language models accept misleading endorsements more readily from higher-expertise persona sources, and the effect can be partially steered away by subtracting a residual-stream vector.

  29. Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge

    cs.CL 2025-08 conditional novelty 6.0 of 10

    A regression controlling for human-rated quality detects positive self- and family-bias in several LLM judges, including GPT-4o and Claude 3.5 Sonnet.

  30. RecGPT Technical Report

    cs.IR 2025-07 conditional novelty 6.0 of 10

    RecGPT integrates three LLMs into Taobao's recommendation pipeline, turning click-log matching into intent mining, tag-based retrieval, and explanation generation, with claimed online gains of 6.33% CTR and 6.96% clic...

  31. Adaptive Repetition for Mitigating Position Bias in LLM-Based Ranking

    cs.LG 2025-07 conditional novelty 6.0 of 10

    An adaptive early-stopping rule for repeated LLM judgments cuts position-bias mitigation cost by roughly 80 percent while keeping the consensus result.

  32. TransEvalnia: Reasoning-based Evaluation and Ranking of Translations

    cs.CL 2025-07 conditional novelty 6.0 of 10

    TransEvalnia, a reasoning-based LLM prompt pipeline for translation evaluation, matches or outperforms MT-Ranker on most WMT pairs and produces human-approved explanations.

  33. LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Small reward models trained on LitBench reach 78% agreement with upvote-derived human preferences in creative writing, beating all zero-shot LLM judges tested.

  34. Time To Impeach LLM-as-a-Judge: Programs are the Future of Evaluation

    cs.LG 2025-06 conditional novelty 6.0 of 10

    LLM-synthesized judging programs, aggregated with weak supervision, can replace direct LLM-as-a-judge scoring at far lower API cost, with better consistency and bias resistance in some settings.

  35. Arbiters of Ambivalence: Challenges of Using LLMs in No-Consensus Tasks

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Across five LLMs and ten no-consensus datasets, neutrality drops sharply when models act as pairwise judges, pointwise judges, or debaters compared to when they generate answers with an explicit neutral option.

  36. syftr: Pareto-Optimal Generative AI

    cs.AI 2025-05 conditional novelty 6.0 of 10

    Syftr applies multi-objective Bayesian optimization to RAG pipeline search, reporting Pareto frontiers on six benchmarks with 6% accuracy gain at equal cost or 37% cost reduction at equal accuracy versus a LlamaIndex default.

  37. Chart-to-Experience: Benchmarking Multimodal LLMs for Predicting Experiential Impact of Charts

    cs.HC 2025-05 conditional novelty 6.0 of 10

    Multimodal LLMs underperform humans at directly rating charts' experiential impact, but they are substantially better at pairwise comparisons, especially when the human ratings differ clearly.

  38. Large Language Models for Predictive Analysis: How Far Are They?

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Existing LLMs perform poorly on predictive analysis, with the best model scoring 24.11/28 and most models failing to generate executable code.

  39. nvAgent: Automated Data Visualization from Natural Language via Collaborative Agent Workflow

    cs.CL 2025-02 conditional novelty 6.0 of 10

    A three-agent workflow (processor, composer, validator) improves automatic data-to-chart generation on the VisEval benchmark, especially for multi-table queries.

  40. Can Large Language Models Understand Preferences in Personalized Recommendation?

    cs.CL 2025-01 conditional novelty 6.0 of 10

    A new grouped-ranking benchmark finds current LLMs score near chance on personalized preference ranking once user rating bias and item quality are controlled.

  41. The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input

    cs.CL 2025-01 conditional novelty 6.0 of 10

    FACTS Grounding is a benchmark and leaderboard that scores LLMs on producing long-form answers fully grounded in up to 32k-token documents, using a validated panel of judge models.

  42. Scaling Point-in-Time Language Models

    cs.CL 2026-04 conditional novelty 5.5 of 10

    Scaling point-in-time LLMs to 4B parameters and 1T temporally filtered tokens narrows the gap to unrestricted models to about 8–11 average points and yields positive out-of-sample Sharpe ratios from news embeddings.

  43. Local Brushstroke Quality Assessment via Vision-Language Feedback

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Multimodal LLMs approximate expert absolute scores for calligraphy brushstrokes but show no significant rank correlation; RAG guidance trades accuracy for ranking.

  44. OPERA: Aligning Open-Ended Reasoning via Objective Perplexity-based Reinforcement Learning

    cs.CL 2026-06 unverdicted novelty 5.0 of 10

    OPERA uses perplexity dynamics as intrinsic rewards for RL alignment on open-ended tasks, synthesizes a 20k trajectory dataset via guiding words and log-prob rollouts, and sets new SOTA on Qwen3-8B matching some propr...

  45. Quantifying and Auditing LLM Evaluation via Positive--Unlabeled Learning

    stat.ML 2026-06 unverdicted novelty 5.0 of 10

    A positive-unlabeled learning approach using partial optimal transport is introduced to audit and correct biases in LLM-as-a-judge systems by aligning limited human positives with unlabeled outputs in embedding space.

  46. Hard Negative Sample-Augmented DPO Post-Training for Small Language Models

    cs.LG 2025-12 unverdicted novelty 5.0 of 10

    A six-dimensional MathVerifier supplies hard negatives and per-sample weights that improve DPO performance on math reasoning for a 1.5B Qwen2.5 model over standard SFT and unweighted DPO.

  47. Contrastive Decoding Mitigates Score Range Bias in LLM-as-a-Judge

    cs.CL 2025-10 unverdicted novelty 5.0 of 10

    Contrastive decoding reduces score-range sensitivity in LLM judges for summarization and raises average Spearman correlation with human ratings by up to 11.7 percent across different score ranges.

  48. Breaking the Mirror: Activation-Based Mitigation of Self-Preference in LLM Evaluators

    cs.CL 2025-09 conditional novelty 5.0 of 10

    Steering vectors flip most unjustified self-preference decisions of an LLM judge but also disturb legitimate ones, showing the bias is not captured by a single linear direction.

  49. Can You Trick the Grader? Adversarial Persuasion of LLM Judges

    cs.CL 2025-08 conditional novelty 5.0 of 10

    Strategically inserted persuasive sentences inflate LLM judges' scores for incorrect math solutions across six benchmarks and fourteen models, but the study lacks length-matched controls separating rhetoric from lengt...

  50. Against 'softmaxing' culture

    cs.HC 2025-06 unverdicted novelty 5.0 of 10

    A position paper arguing that AI evaluations should shift from defining culture to understanding when culture becomes relationally valid.

  51. A Cross-Cultural Comparison of LLM-based Public Opinion Simulation: Evaluating Chinese and U.S. Models on Diverse Societies

    cs.CL 2025-06 conditional novelty 5.0 of 10

    Across U.S. and Chinese survey questions, DeepSeek, GPT-4o, Qwen2.5, and Llama-3.3 all show demographic overgeneralization, with no consistent home-field advantage for the Chinese model.

  52. An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliability

    cs.CL 2025-06 conditional novelty 5.0 of 10

    The reliability of LLM-as-a-Judge depends strongly on scoring rubrics and reference answers; sampling with averaging outperforms greedy decoding, and chain-of-thought reasoning adds little when rubrics are clear.

  53. Debate-to-Detect: Reformulating Misinformation Detection as a Real-World Debate with Large Language Models

    cs.CL 2025-05 conditional novelty 5.0 of 10

    A structured multi-agent debate framework with domain-specialized AI agents and a five-dimension scoring rubric improves LLM-based fake news detection by several F1 points.

  54. Unbiased Evaluation of Large Language Models from a Causal Perspective

    cs.AI 2025-02 reject novelty 5.0 of 10

    The paper argues that perturbing benchmark questions with rule-based interventions gives a less contaminated, more interpretable evaluation of LLMs than static benchmarks or agent-generated questions.

  55. Creating Multilingual Mental Health Dialogue Datasets: Limits of Persona-Based Localization via Nationality and Language

    cs.CL 2026-06 unverdicted novelty 4.0 of 10

    Modifying nationality and language parameters in English-centric personas for mental health dialogues introduces clinical inconsistencies across languages and causes LLM judges to perform inaccurately on non-English d...

  56. MimirRAG: A Multi-Agent RAG Framework for Financial Data Retrieval with Metadata Integration

    cs.LG 2026-05 unverdicted novelty 4.0 of 10

    MimirRAG, a multi-agent RAG framework with metadata integration and table-aware chunking, reaches 89.3% accuracy on FinanceBench and outperforms prior baselines for financial document retrieval.

  57. AI for Auto-Research: Roadmap & User Guide

    cs.AI 2026-05 unverdicted novelty 4.0 of 10

    The paper delivers a stage-by-stage roadmap for AI in research, showing reliable assistance in retrieval and tool tasks but fragility in novelty and judgment, advocating human-governed collaboration.

  58. Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation

    cs.LG 2026-02 conditional novelty 4.0 of 10

    A taxonomy-driven survey arguing that reward design is the central mechanism shaping reliable LLM reasoning, with maps of reward paradigms, reward-hacking failure modes, and benchmark pitfalls.

  59. TimeMKG: Knowledge-Infused Causal Reasoning for Multivariate Time Series Modeling

    cs.LG 2025-08 unverdicted novelty 4.0 of 10

    TimeMKG injects LLM-generated knowledge graphs about variable semantics into multivariate time series models to improve forecasting and classification.

  60. Data and AI governance: Promoting equity, ethics, and fairness in large language models

    cs.CL 2025-08 unverdicted novelty 4.0 of 10

    The paper proposes a lifecycle governance framework, built on the authors' BEATS benchmark, to quantify and mitigate bias, ethics, fairness, and factuality failures in large language models.

See all 69 Pith citations

Reference graph

Works this paper leans on

25 extracted references · 25 canonical work pages · cited by 69 Pith papers (see all)

  1. [1]

    arXiv preprint arXiv:2307.03025 , year=

    URL https://arxiv.org/abs/2307.03025. Siyuan Wu, Yue Huang, Chujie Gao, Dongping Chen, Qihui Zhang, Yao Wan, Tianyi Zhou, Xian- gliang Zhang, Jianfeng Gao, Chaowei Xiao, et al. Unigen: A unified framework for textual dataset generation using large language models. arXiv preprint arXiv:2406.18966, 2024a. Yuanwei Wu, Yue Huang, Yixin Liu, Xiang Li, Pan Zhou...

    work page arXiv 2024

  2. [2]

    {number}% of people believe that {chosen_model} is better in this question

    include fallacy-oversight bias, authority bias, and beauty bias. Recognizing these biases is essential for developing more objective and trustworthy LLM evaluation methods. B D ETAILS OF BIAS TYPES ▷ Position bias: LLMs may favor responses based on their position in the input. This bias affects how the model processes information, and following Zheng et a...

    work page 2024

  3. [3]

    Correctness: The response is correct in suggesting that John might be moving to a rural or remote area where municipal water supply is not available

    work page

  4. [4]

    Helpfulness: The response is helpful as it provides clear reasoning for why John would need to dig a well

    work page

  5. [5]

    Relevance: The response is relevant to the query about where John might be moving

    work page

  6. [6]

    Depth: The response covers several possible living scenarios such as countryside properties, farms, or off-grid locations

    work page

  7. [7]

    Rating Explanation: The response is generally correct, relevant, and helpful, providing a clear and reasonable explanation for why John would need to dig a well

    Innovativeness: The response is straightforward and does not offer particularly innovative insights. Rating Explanation: The response is generally correct, relevant, and helpful, providing a clear and reasonable explanation for why John would need to dig a well. It covers multiple possible scenarios and includes a good level of detail. However, it lacks a...

    work page

  8. [8]

    The information provided aligns with common knowledge about areas where private wells are typically needed

    Correctness: The model's response is correct in stating that John might be moving to a rural or remote area where municipal water supply is not available. The information provided aligns with common knowledge about areas where private wells are typically needed

    work page

Show all 25 references
  1. [9]

    It also elaborates on the need for self-sufficiency in such areas

    Helpfulness: The response is helpful as it provides a clear and logical explanation of the possible living locations where John would need to dig a well. It also elaborates on the need for self-sufficiency in such areas. 3.Relevance: The response is relevant to the user's quer...

  2. [10]

    Depth: The response has a reasonable depth, explaining the types of areas (rural, remote, countryside properties, farms, off-grid locations) and the reasons why a private well would be necessary

  3. [11]

    [[A]]" if assistant A is better,

    Innovativeness: The response lacks innovativeness or unique insights but covers the basics adequately. Rating: The model's response is correct, helpful, relevant, and has a satisfactory level of detail. However, it does not exceed the reference answer in quality and lacks any ...

    1990

  4. [12]

    Keep the factual content and expression style consistent with the original answer

  5. [13]

    Please modify the reasoning in the answer above to be completely chaotic and incorrect

  6. [15]

    [[A]]" if {model_a} is better,

    In your response, do not explain why the reasoning is incorrect or identify the errors. Figure 19: Prompt template for expanding sentence length. 28 Preprint Prompt Template: Compassion-Fade [System] [System] Please act as an impartial judge and evaluate the quality of the res...

    1990

  7. [16]

    Keep the length with the original answer

  8. [17]

    Ensure that the conclusion of the answer remains unchanged

  9. [18]

    [[A]]" if assistant A is better,

    Make the answer style {emotion}. """ Figure 27: Prompt template for sentiment bias evaluation. 32 Preprint Prompt Template: Diversity Bias Evaluation [System] Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user q...

  10. [19]

    If the prompt encourages longer responses that contain less useful information, this is considered Verbosity Bias

    Verbosity Bias : Language models may judge responses more favorably if they are longer. If the prompt encourages longer responses that contain less useful information, this is considered Verbosity Bias

  11. [20]

    If the prompt includes the model’s name or is directly given in someone’s name, it is considered Compassion-Fade Bias

    Compassion-Fade Bias : Language models’ judgments may be influenced by the name of the model being judged or anonymization strategies. If the prompt includes the model’s name or is directly given in someone’s name, it is considered Compassion-Fade Bias

  12. [21]

    many people prefer answer X

    Bandwagon-Effect Bias : Language models’ judgments may be influenced by majority opinions. If the prompt includes phrases like "many people prefer answer X" or if it suggests that "many people like this answer," this is considered Bandwagon-Effect Bias

  13. [22]

    If the prompt contains irrelevant information, it is considered Distraction Bias

    Distraction Bias: Language models’ judgments may be impacted by introduced distractions, especially when evaluating high-quality and low-quality content. If the prompt contains irrelevant information, it is considered Distraction Bias

  14. [23]

    If the prompt allows for responses that contain clear logical fallacies but still lead to a correct result, this is considered Fallacy-Oversight Bias

    Fallacy-Oversight Bias : Language models may overlook logical fallacies during evaluation. If the prompt allows for responses that contain clear logical fallacies but still lead to a correct result, this is considered Fallacy-Oversight Bias

  15. [24]

    If the prompt encourages responses that contain cited information that might be false, it is considered Authority Bias

    Authority Bias : Language models’ judgments may be influenced by authoritative sources such as book citations, website references, or quotes from famous individuals. If the prompt encourages responses that contain cited information that might be false, it is considered Authority Bias

  16. [25]

    If the prompt encourages responses with obvious emotional expressions such as Cheerful, Sad, Angry, or Fear, it is considered Sentiment Bias

    Sentiment Bias : Language models may prefer certain emotional tones, leading to biases based on emotional expression rather than content quality. If the prompt encourages responses with obvious emotional expressions such as Cheerful, Sad, Angry, or Fear, it is considered Senti...

  17. [26]

    If the prompt mentions belonging to any of these or similar identities, it is considered Diversity Bias

    Diversity Bias: Language models’ judgments may be affected by the identity categories involved (e.g., Female, Black individuals, Homosexuals, Muslims, Refugees, HIV patients). If the prompt mentions belonging to any of these or similar identities, it is considered Diversity Bi...

Pith tools

Reviewed May 15, 2026 · model on record in the stance chip above.