Pith. sign in

REVIEW 3 major objections 5 minor 75 references

This paper claims that an LLM's pairwise-judging logic can be distilled once into a committee of small Python programs, and that this committee matches a 13-billion-parameter LLM judge in accuracy while running tens of times faster and cutt

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 12:45 UTC pith:3AGLHYVS

load-bearing objection A well-engineered systems paper with a plausible core idea, but the headline 'match' claim is only demonstrated on filtered, easier subsets and the abstract overstates the cost savings; deserves serious refereeing. the 3 major comments →

arxiv 2607.22561 v1 pith:3AGLHYVS submitted 2026-05-29 cs.AI cs.LG

Codifying the Judge: Scalable Evaluation via Program Distillation

Tzu-Heng Huang , Shengqi Qiu , Frederic Sala This is my paper
classification cs.AI cs.LG
keywords LLM-as-a-judgeprogram distillationprogrammatic judgesPAJAMAweak supervisionLLM routingreward model distillationpreference evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that the decision logic of an LLM used as a judge can be moved out of the model entirely. Instead of calling an LLM on every candidate pair, the authors have an LLM write Python programs that encode specific evaluation rubrics; a calibrated committee of fewer than twenty such programs then scores responses directly. On five preference datasets, the committee's accuracy (about 78.1%) lands close to a 13-billion-parameter open-weight LLM judge (about 79.7%) while judging roughly 47 times faster, and low-confidence cases can be routed to an LLM to push the accuracy-throughput frontier outward. The same program labels train a reward model that generalizes as well as or better than labels from a proprietary LLM at about 50x lower API cost, and the programs can be audited and patched to remove biases. The paper is explicit that its evaluation sets exclude coding and math prompts and drop ambiguous or tied cases, so the broad claim is bounded by that filtering.

Core claim

The core discovery is that LLM-as-a-judge does not have to mean per-sample LLM inference. By asking the LLM once to synthesize executable Python judging functions from a set of ten curated rubrics, and then calibrating, selecting, and aggregating those programs into a committee, the authors obtain a judge whose average accuracy on five preference datasets is 78.11%, within about 1.6 points of a 13B open-weight LLM judge, while running at 439 samples/second compared to 9.3. A committee of fewer than twenty programs suffices; when the committee abstains or is low-confidence, a router sends the sample to an LLM, and the hybrid system improves both accuracy and throughput across twelve tested LL

What carries the argument

The central object is PAJAMA, a committee-of-programs judging system. It works by (1) prompting an LLM to write Python judging functions for ten distinct rubrics, (2) calibrating each program's raw scores with per-program abstention thresholds and selecting the top programs by validation accuracy, (3) aggregating their votes with a generative label model that learns each program's reliability from agreement and disagreement patterns, and (4) routing samples where the committee abstains or its posterior is near 0.5 to an LLM judge. The load-bearing mechanism is the committee's disagreement: individual programs are noisy and repetitive, but their vote variance and the aggregator's posterior co

Load-bearing premise

The load-bearing premise is that the filtered evaluation subsets—where ties, ambiguous labels, and low-confidence or coding/math prompts are removed—are representative enough of real LLM-judging workloads that matching a 13-billion-parameter judge on them establishes the general claim.

What would settle it

Re-run the same committee on the unfiltered evaluation data, keeping ties, ambiguous labels, and coding/math prompts, and compare accuracy and coverage against the 13-billion-parameter judge; if accuracy collapses or coverage drops well below the reported ~95%, the central 'match a 13B judge' claim fails outside the filtered regime.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Evaluation cost becomes a one-time synthesis bill: once programs exist, every additional sample is judged locally, so cost no longer grows with dataset size.
  • The committee's disagreement provides a free confidence signal: hybrid systems can spend expensive LLM calls only on the hard fraction of pairs and keep most of the LLM's accuracy at a fraction of the latency.
  • Reward-model distillation shifts from buying labels per sample to buying a fixed set of programs; at scale this gap widens linearly in the number of preference pairs.
  • Bias becomes patchable: since judging logic is code, a detected bias can be edited or re-calibrated by a coding agent, and position bias is structurally absent because the programs score each response independently.
  • Transparency is not just a side benefit: the same inspectable logic makes low-confidence routing and selective escalation possible in the first place.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The authors leave implicit that the same pattern—use a cheap, transparent committee to screen and an expensive model only for the residual—could apply beyond preference judging, e.g., to rubric scoring, safety filtering, or data curation, wherever an LLM currently scores every item.
  • If the filtering is as important as it appears, the method's true boundary is linguistic preference evaluation; a natural test would be to synthesize programs for code or math that call execution or symbolic checkers rather than heuristics, and measure how much routing to the LLM is then needed.
  • The routing-threshold sweeps in the paper are traced after seeing test-set results; a stricter practical evaluation would freeze thresholds on validation data and measure the frontier, which the authors do not report.
  • The bias-calibration result suggests a cheap, repeatable pipeline for debiasing judges: generate, audit with a coding agent, re-run the bias probes, and repeat—something opaque model judges do not offer.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes PAJAMA, a system that distills LLM judging logic into a committee of executable Python programs. Programs are synthesized once from curated rubrics, calibrated on a validation set, selected by validation accuracy, and aggregated via a weak-supervision label model. Uncertain cases are routed to an LLM judge. The authors claim that (C1) programmatic judges match the accuracy of OLMO-2-13B-INSTRUCT on five preference datasets while running ~47× faster; (C2) program-derived routing signals advance the accuracy–throughput Pareto frontier; (C3) reward models distilled from program labels outperform GPT-4-labeled reward models on RewardBench at 45–50× lower API cost; and (C4) programmatic judges are robust to common biases and can be improved by code-agent calibration.

Significance. If the central claims hold, program distillation is a genuinely useful alternative to per-sample LLM judging: it replaces repeated inference with a one-time synthesis step, yields inspectable and editable decision logic, and produces cheap supervision for reward modeling. The paper is honest about several limitations and includes useful ablations (e.g., validation-set size, coverage rates). The availability of source code and detailed prompts strengthens reproducibility. However, the headline claims are broader than what the experiments actually support, mainly because of the filtering procedure and the test-set-swept routing analysis.

major comments (3)
  1. [Appendix B / §4.1] The headline claim that programmatic judges 'match' OLMO-2-13B across five datasets is supported only on a filtered subset. Appendix B removes ties, low-confidence/ambiguous examples, and all coding and math prompts; the same filtered distribution is used for threshold tuning, top-k selection, and aggregation (§3.3). The abstract and §4.1 state 'across five datasets' without this qualification, and the limitation section (Appendix E) admits programs 'may fail to generalize' to math/coding. Please report results on the unfiltered sets, or at least include math/coding subsets, and qualify the claim accordingly.
  2. [§4.2 / Figures 4–5] The Pareto-frontier claim depends on sweeping the escalation threshold after seeing test-set accuracy. The text says 'sweeping the threshold traces the full hybrid accuracy–throughput curve' but gives no evidence that the threshold is chosen on a validation split or held out. As plotted, the frontier is an envelope over test-set-optimal operating points, so the '+5% accuracy at 2.9× throughput' example is not a reproducible operating point. Report a fixed threshold chosen on validation (or a curve with confidence bands) before claiming PAJAMA 'improves both accuracy and throughput.'
  3. [§4.3 / Table 1] The abstract's claim that a reward model distilled from programs 'outperforms' one trained on GPT-4 labels is based on the RewardBench average, but the category breakdown is mixed. On Chat Hard, PAJAMA is 30.04 vs GPT-4's 38.82 when trained on Prometheus, and 44.52 vs 38.82 on JudgeLM; only the Chat and Reasoning categories drive the average. Please report the per-category comparison prominently, add variability estimates (e.g., seeds), and soften or correct the claim to reflect the actual deficit on Chat Hard.
minor comments (5)
  1. [Figure 1] 'TF-vector cosine similarity' appears to be a typo for 'TF-IDF vector cosine similarity'. Please clarify.
  2. [§4.1 / Table 3] §4.1 says 'hold out an additional 500 examples as a validation set' for each dataset, but Table 3 reports only 170 validation examples for MultiPref. This inconsistency affects reproducibility.
  3. [§4.2 / Figure 4] Please define the 'escalation budget' and how the threshold maps to the x-axis (throughput). Currently the dominance claim is hard to interpret because throughput is a function of the routed fraction and the LLM's serving speed.
  4. [Table 2] The 'Δ(Reduction)' row shows +4.36 for Rich Content BWR, meaning PAJAMA (Calibrated) actually increases bias win rate on that category, yet the text says 'both flip rate and bias win rate can be substantially reduced.' Please correct the sentence or explain the discrepancy.
  5. [General] Minor language issues: 'Pajama's fallback' should be 'PAJAMA's fallback'; 'delivers competitive' should be 'deliver competitive'; the abstract's '47.25× faster' and §4.3's '50× lower cost' should reference the exact tables for clarity.

Circularity Check

0 steps flagged

No significant circularity: the central claims are held-out empirical results; minor caveats (filtered test sets, test-swept routing frontier) affect external validity rather than demonstrating self-referential derivation.

full rationale

The paper's derivation chain is empirical, not equation-level. Programmatic judges are synthesized once, thresholds are tuned on a 500-example validation split, and top-k selection plus aggregator weights are fit on that same split, while reported accuracies come from held-out test sets (Sec 4.1 and Appendix C: 'For each program f_j, we search over a grid of candidate thresholds ... and select the one that maximizes validation accuracy'). This is standard model selection, not fitting-then-predicting on the same data. The weak-supervision aggregator is attributed to external Snorkel work plus the authors' own extensions, but no load-bearing uniqueness or ansatz claim is imported from the self-citations; the aggregation step is described operationally as a standard label model. The two validity caveats - Appendix B's filtering that drops ties, low-confidence examples, and math/coding prompts, and Sec 4.2's 'Sweeping the threshold traces the full hybrid accuracy-throughput curve' - are transparently disclosed and affect how broadly the 'match' claim generalizes; they do not make a reported prediction equal to a fitted input by construction. The paper even states the scope restriction in its Limitations: 'For complex reasoning tasks such as mathematics or coding, synthesized programs may fail to generalize.' Because the central comparison is held-out and benchmarked against external LLM judges, there is no significant circularity; score 2 reflects minor self-citation presence and the filtered/test-swept evaluation concerns, not definitional circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central claim rests on validation labels, one-time proprietary LLM synthesis, and the unstated normalization regime; the main free parameters are thresholds, committee size, aggregator weights, routing threshold, and the dataset-narrowing filters.

free parameters (5)
  • Per-program vote threshold tau_j = grid 0.00-0.14, selected per program on validation
    Converts quality difference into +1/-1/0 vote; tuned to maximize validation accuracy (Section 3.3, Appendix C).
  • Top-k selected programs = 8-21 per dataset, e.g. 8 on Prometheus, 21 on JudgeLM
    Programs kept after validation-accuracy selection; k is a data-dependent hyperparameter (Section 3.3, Table 4).
  • Snorkel label model aggregation weights = learned on validation preference matrix
    Weights combine program votes into a final verdict; fit to validation labels (Section 3.3).
  • Escalation threshold for routing = swept; not fixed
    Controls the fraction of samples escalated to an LLM; in Section 4.2 it is swept to trace the full Pareto curve, and the paper does not state that it is set on validation.
  • Data filtering thresholds = drop ties/ambiguous/low-confidence; exclude math/coding; minimum score-gap
    Ad hoc dataset-narrowing choices described in Appendix B; they directly affect the difficulty of the evaluation and the magnitude of the reported gains.
axioms (5)
  • domain assumption Validation ground-truth labels (GPT-4 or human) are reliable enough to tune programs
    Program synthesis few-shot examples, threshold tuning, top-k selection, and aggregator training all depend on validation labels; if those labels are biased, PAJAMA inherits the bias.
  • domain assumption Programs synthesized one-time by Claude Opus 4.6 generalize to unseen test inputs
    The entire approach assumes a single synthesis step produces reusable judging logic; no formal guarantee or stability analysis is provided.
  • domain assumption Min-max normalization does not use test-set statistics
    Section 3.3 says outputs are min-max normalized but does not state whether the min/max are computed on validation or test; if computed on test, the evaluation leaks.
  • standard math Weak supervision label-model assumptions hold for program votes
    Snorkel-style aggregation assumes labelers are conditionally independent given the true label; the paper applies it to program verdicts without testing this assumption.
  • ad hoc to paper Filtering out coding and math leaves a meaningful preference benchmark
    Appendix B excludes these domains because they are 'outside the scope of rubric-based linguistic evaluation', which narrows the scope of the central claim.

pith-pipeline@v1.3.0-alltime-deepseek · 18171 in / 12909 out tokens · 131957 ms · 2026-08-02T12:45:11.779358+00:00 · methodology

0 comments
read the original abstract

LLM-as-a-judge has become the standard for automated evaluation, but it suffers from high cost, significant latency, and opaque decisions -- limitations that undermine its scalability and reliability. We address these with a simple, efficient alternative: program distillation. Instead of prompting an LLM at the evaluation time, we distill its decision logic into a committee of programs that score candidates directly. These programmatic judges offer transparency, are easily inspected or edited, and eliminate per-sample API costs. Building on this notion, we introduce PAJAMA, a system that synthesizes programs as judges, aggregates their decisions into a joint verdict, and incorporates a fallback mechanism to selectively escalate low-confidence cases to an LLM. Across five datasets and four model families, we show that programmatic judges can match the performance of a 13B-size LLM judge. When using program outputs as routing signals, PAJAMA improves both accuracy and throughput and advances the Pareto frontier. Beyond evaluation, programmatic judges produce cheap and effective reward signals: on RewardBench, a reward model distilled from programs' verdicts outperforms one trained on a proprietary LLM's labels at two orders of magnitude lower API cost.

Figures

Figures reproduced from arXiv: 2607.22561 by Frederic Sala, Shengqi Qiu, Tzu-Heng Huang.

Figure 1
Figure 1. Figure 1: An example of synthesizing programmatic judges. Given a synthesis prompt (top-left), an LLM generates a Python judging_function that articulates an evaluation rubric—for example, logical coherence—using interpretable features such as semantic drift via TF-vector cosine similarity or topic consistency across discourse segments. These features are combined into a weighted score, with explicit penalties for c… view at source ↗
Figure 2
Figure 2. Figure 2: PAJAMA Workflow. Given a query q and two candidate responses r (1),r(2), a diverse pool of programmatic judges—synthesized by an LLM from curated rubrics—produces initial evaluations. These program outputs are calibrated and selected, and their verdicts are aggregated into a combined decision. Uncertain cases are then routed to an LLM judge to produce the final preference. 3 Framework We begin with an over… view at source ↗
Figure 3
Figure 3. Figure 3: Effectiveness of Programmatic Judges. Accuracy vs. throughput across five preference datasets. Each subplot title reports the number of programs retained after selection. Programmatic judges achieve extremely high throughput while remaining competitive in accuracy with mid-sized LLM judges—matching OLMO-2-13B-INSTRUCT on average. C2. Routing and Pareto Frontier. Program-derived signals serve as reliable ro… view at source ↗
Figure 4
Figure 4. Figure 4: Routing within the OLMo-2 family. Accuracy vs. throughput as the escalation threshold is swept across different routing signals. Endpoints represent pure-program (right) and pure-LLM (left) evaluation. PAJAMA-derived signals (aggregator posterior, vote variance) consistently dominate other model-free routing methods across all model sizes. accuracy against ground-truth preference labels, and throughput, th… view at source ↗
Figure 5
Figure 5. Figure 5: Hybrid evaluation pushes the Pareto frontier further. Each panel shows accuracy vs. throughput as PAJAMA routes uncertain pairs to one of 12 LLM judges. Dashed curves trace each judge’s hybrid trajectory; the red envelope is the resulting PAJAMA frontier, while the gray envelope is the LLM-only one from §4.1. which routes a uniformly sampled fraction of pairs. Each baseline is swept over the same escalatio… view at source ↗
Figure 6
Figure 6. Figure 6: Routing within the Qwen2.5 family [PITH_FULL_IMAGE:figures/full_fig_p022_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Routing within the Gemma-3 family [PITH_FULL_IMAGE:figures/full_fig_p022_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

75 extracted references · 16 linked inside Pith

  1. [1]

    The Innovation2024,

    Gu, J.; Jiang, X.; Shi, Z.; Tan, H.; Zhai, X.; Xu, C.; Li, W.; Shen, Y .; Ma, S.; Liu, H.; others A survey on llm-as-a-judge. The Innovation2024,

  2. [2]

    Llms-as-judges: a comprehensive survey on llm-based evaluation methods.arXiv preprint arXiv:2412.055792024,

    Li, H.; Dong, Q.; Chen, J.; Su, H.; Zhou, Y .; Ai, Q.; Ye, Z.; Liu, Y . Llms-as-judges: a comprehensive survey on llm-based evaluation methods.arXiv preprint arXiv:2412.055792024,

    Pith/arXiv arXiv

  3. [3]

    JudgeLM: Fine-tuned Large Language Models are Scalable Judges

    Zhu, L.; Wang, X.; Wang, X. JudgeLM: Fine-tuned Large Language Models are Scalable Judges. The Thirteenth International Conference on Learning Representations. 2025

    2025

  4. [4]

    Prometheus: Inducing Fine-Grained Evaluation Capability in Language Models

    Kim, S.; Shin, J.; Cho, Y .; Jang, J.; Longpre, S.; Lee, H.; Yun, S.; Shin, S.; Kim, S.; Thorne, J.; Seo, M. Prometheus: Inducing Fine-Grained Evaluation Capability in Language Models. The Twelfth International Conference on Learning Representations. 2024

    2024

  5. [5]

    PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization

    Wang, Y .; Yu, Z.; Yao, W.; Zeng, Z.; Yang, L.; Wang, C.; Chen, H.; Jiang, C.; Xie, R.; Wang, J.; Xie, X.; Ye, W.; Zhang, S.; Zhang, Y . PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization. The Twelfth International Conference on Learning Representations. 2024

    2024

  6. [6]

    Learning LLM-as-a-Judge for Preference Alignment

    Ye, Z.; Li, X.; Li, Q.; Ai, Q.; Zhou, Y .; Shen, W.; Yan, D.; LIU, Y . Learning LLM-as-a-Judge for Preference Alignment. The Thirteenth International Conference on Learning Representations. 2025

    2025

  7. [7]

    E.; Sukhbaatar, S

    Wu, T.; Yuan, W.; Golovneva, O.; Xu, J.; Tian, Y .; Jiao, J.; Weston, J. E.; Sukhbaatar, S. Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Suzhou, China, 2025; pp 11537–11554

    2025

  8. [8]

    Y .; Fazel-Zarandi, M.; Weston, J.; Li, X

    Wang, T.; Kulikov, I.; Golovneva, O.; Yu, P.; Yuan, W.; Dwivedi-Yu, J.; Pang, R. Y .; Fazel-Zarandi, M.; Weston, J.; Li, X. Self-taught evaluators.arXiv preprint arXiv:2408.026662024,

    Pith/arXiv arXiv

  9. [9]

    Gunjal, A.; Wang, A.; Lau, E.; Nath, V .; He, Y .; Liu, B.; Hendryx, S. M. Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. The Fourteenth International Conference on Learning Representations. 2026

    2026

  10. [10]

    RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioning.arXiv preprint arXiv:2603.091602026,

    Huang, T.-H.; Salekin, S.; Movellan, J.; Sala, F.; Bilkhu, M. RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioning.arXiv preprint arXiv:2603.091602026,

    arXiv

  11. [11]

    Z.; Ivison, H.; Kishore, V .; Zhuo, J.; Zhao, X.; Park, M.; Finlayson, S

    Shao, R.; Asai, A.; Shen, S. Z.; Ivison, H.; Kishore, V .; Zhuo, J.; Zhao, X.; Park, M.; Finlayson, S. G.; Sontag, D.; others Dr tulu: Reinforcement learning with evolving rubrics for deep research.arXiv preprint arXiv:2511.193992025,

    Pith/arXiv arXiv

  12. [12]

    Singh, A.; Fry, A.; Perelman, A.; Tart, A.; Ganesh, A.; El-Kishky, A.; McLaughlin, A.; Low, A.; Ostrow, A.; Ananthram, A.; others Openai gpt-5 system card.arXiv preprint arXiv:2601.032672025,

    Pith/arXiv arXiv

  13. [13]

    Google DeepMind Gemini 3 Pro Model Card. 2025

    2025

  14. [14]

    Tuning LLM Judge Design Decisions for 1/1000 of the Cost

    Salinas, D.; Swelam, O.; Hutter, F. Tuning LLM Judge Design Decisions for 1/1000 of the Cost. Forty-second International Conference on Machine Learning. 2025

    2025

  15. [15]

    One token to fool llm-as-a-judge.arXiv preprint arXiv:2507.087942025,

    Zhao, Y .; Liu, H.; Yu, D.; Kung, S.; Chen, M.; Mi, H.; Yu, D. One token to fool llm-as-a-judge.arXiv preprint arXiv:2507.087942025,

    Pith/arXiv arXiv

  16. [16]

    Investigating the Vulnerability of LLM-as-a-Judge Architectures to Prompt-Injection Attacks.International Journal of Open Information Technologies2025,13, 1–6

    Maloyan, N.; Ashinov, B.; Namiot, D. Investigating the Vulnerability of LLM-as-a-Judge Architectures to Prompt-Injection Attacks.International Journal of Open Information Technologies2025,13, 1–6. 11

  17. [17]

    H.; Chen, S.; Liu, Z.; Jiang, F.; Wang, B

    Chen, G. H.; Chen, S.; Liu, Z.; Jiang, F.; Wang, B. Humans or LLMs as the Judge? A Study on Judgement Bias. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Miami, Florida, USA, 2024; pp 8301–8327

    2024

  18. [18]

    V .; Zhang, X

    Ye, J.; Wang, Y .; Huang, Y .; Chen, D.; Zhang, Q.; Moniz, N.; Gao, T.; Geyer, W.; Huang, C.; Chen, P.-Y .; Chawla, N. V .; Zhang, X. Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge. The Thirteenth International Conference on Learning Representations. 2025

    2025

  19. [19]

    Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge

    Shi, L.; Ma, C.; Liang, W.; Diao, X.; Ma, W.; V osoughi, S. Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge. Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Con- ference of the Asia-Pacific Chapter of the Association for Computational Linguistics. Mumbai, India, 2025; pp 292–314

    2025

  20. [20]

    Can you trust llm judgments? reliability of llm-as-a-judge.arXiv preprint arXiv:2412.125092024,

    Schroeder, K.; Wood-Doughty, Z. Can you trust llm judgments? reliability of llm-as-a-judge.arXiv preprint arXiv:2412.125092024,

    Pith/arXiv arXiv

  21. [21]

    Zhao, J.; Shin, C.; Huang, T.-H.; GNVV , S. S. S. N.; Sala, F. CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation.arXiv preprint arXiv:2603.000392026,

    arXiv

  22. [22]

    Walsh, E. P. et al. 2 OLMo 2 Furious (COLM’s Version). Second Conference on Language Modeling. 2025

    2025

  23. [23]

    Yang, Q. A. et al. Qwen2.5 Technical Report.ArXiv2024,abs/2412.15115

    Pith/arXiv arXiv

  24. [24]

    Y .; Chandu, K.; Dziri, N.; Kumar, S.; Zick, T.; Choi, Y .; Smith, N

    Lambert, N.; Pyatkin, V .; Morrison, J.; Miranda, L.; Lin, B. Y .; Chandu, K.; Dziri, N.; Kumar, S.; Zick, T.; Choi, Y .; Smith, N. A.; Hajishirzi, H. RewardBench: Evaluating Reward Models for Language Modeling. Findings of the Association for Computational Linguistics: NAACL 2025. Albuquerque, New Mexico, 2025; pp 1755–1797

    2025

  25. [25]

    Can Large Language Models Be an Alternative to Human Evaluations? Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers)

    Chiang, C.-H.; Lee, H.-y. Can Large Language Models Be an Alternative to Human Evaluations? Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers). Toronto, Canada, 2023; pp 15607–15631

    2023

  26. [26]

    Length-Controlled AlpacaEval: A Simple Debiasing of Automatic Evaluators

    Dubois, Y .; Liang, P.; Hashimoto, T. Length-Controlled AlpacaEval: A Simple Debiasing of Automatic Evaluators. First Conference on Language Modeling. 2024

    2024

  27. [27]

    Miranda, L. J. V .; Wang, Y .; Elazar, Y .; Kumar, S.; Pyatkin, V .; Brahman, F.; Smith, N. A.; Hajishirzi, H.; Dasigi, P. Hybrid preferences: Learning to route instances for human vs. AI feedback. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers). 2025; pp 7162–7200

    2025

  28. [28]

    Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; others Training language models to follow instructions with human feedback.Advances in neural information processing systems2022,35, 27730–27744

  29. [29]

    Bai, Y . et al. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback.ArXiv 2022,abs/2204.05862

    Pith/arXiv arXiv 2022

  30. [30]

    J.; Choi, E

    Wang, V .; Zhang, M. J.; Choi, E. Improving LLM-as-a-Judge Inference with the Judgment Distribution. Findings of the Association for Computational Linguistics: EMNLP 2025. Suzhou, China, 2025; pp 23173–23199

    2025

  31. [31]

    H.; Ehrenberg, H.; Fries, J.; Wu, S.; Ré, C

    Ratner, A.; Bach, S. H.; Ehrenberg, H.; Fries, J.; Wu, S.; Ré, C. Snorkel: Rapid training data creation with weak supervision. Proceedings of the VLDB endowment. International conference on very large data bases. 2017; p 269

    2017

  32. [32]

    J.; De Sa, C

    Ratner, A. J.; De Sa, C. M.; Wu, S.; Selsam, D.; Ré, C. Data programming: Creating large training sets, quickly.Advances in neural information processing systems2016,29. 12

  33. [33]

    Training complex models with multi-task weak supervision

    Ratner, A.; Hancock, B.; Dunnmon, J.; Sala, F.; Pandey, S.; Ré, C. Training complex models with multi-task weak supervision. Proceedings of the AAAI conference on artificial intelligence. 2019; pp 4763–4771

    2019

  34. [34]

    C.; Sala, F

    Shin, C.; Li, W.; Vishwakarma, H.; Roberts, N. C.; Sala, F. Universalizing Weak Supervision. International Conference on Learning Representations. 2022

    2022

  35. [35]

    Snorkel metal: Weak supervision for multi-task learning

    Ratner, A.; Hancock, B.; Dunnmon, J.; Goldman, R.; Ré, C. Snorkel metal: Weak supervision for multi-task learning. Proceedings of the Second Workshop on Data Management for End-To-End Machine Learning. 2018; pp 1–4

    2018

  36. [36]

    The ALCHEmist: Automated Labeling 500x CHEaper than LLM Data Annotators

    Huang, T.-H.; Cao, C.; Bhargava, V .; Sala, F. The ALCHEmist: Automated Labeling 500x CHEaper than LLM Data Annotators. The Thirty-eighth Annual Conference on Neural Information Processing Systems. 2024

    2024

  37. [37]

    Scriptoriumws: A code generation assistant for weak supervision.arXiv preprint arXiv:2502.123662025,

    Huang, T.-H.; Cao, C.; Schoenberg, S.; Vishwakarma, H.; Roberts, N.; Sala, F. Scriptoriumws: A code generation assistant for weak supervision.arXiv preprint arXiv:2502.123662025,

    Pith/arXiv arXiv

  38. [38]

    Autows-bench-101: Benchmarking automated weak supervision with 100 labels.Advances in Neural Information Processing Systems2022,35, 8912–8925

    Roberts, N.; Li, X.; Huang, T.-H.; Adila, D.; Schoenberg, S.; Liu, C.-Y .; Pick, L.; Ma, H.; Albarghouthi, A.; Sala, F. Autows-bench-101: Benchmarking automated weak supervision with 100 labels.Advances in Neural Information Processing Systems2022,35, 8912–8925

  39. [39]

    J.; Adila, D.; Sala, F

    Huang, T.-H.; Shin, C.; Tay, S. J.; Adila, D.; Sala, F. Multimodal data curation via object detection and filter ensembles. arXiv preprint arXiv:2401.122252024,

    Pith/arXiv arXiv

  40. [40]

    H.; Kellman, P.; Xue, H.; Sala, F.; Langlotz, C.; Re, C

    Hooper, S.; Wornow, M.; Seah, Y . H.; Kellman, P.; Xue, H.; Sala, F.; Langlotz, C.; Re, C. Cut out the annotator, keep the cutout: better segmentation with weak supervision. International Conference on Learning Representations. 2021

    2021

  41. [41]

    F.; Orr, L.; Guha, N.; Bhatia, K.; Chami, I.; Re, C

    Arora, S.; Narayan, A.; Chen, M. F.; Orr, L.; Guha, N.; Bhatia, K.; Chami, I.; Re, C. Ask Me Anything: A simple strategy for prompting language models. The Eleventh International Conference on Learning Representations. 2023

    2023

  42. [42]

    K.; Chen, M

    Saad-Falcon, J.; Buchanan, E. K.; Chen, M. F.; Huang, T.-H.; McLaughlin, B.; Bhathal, T.; Zhu, S.; Athiwaratkun, B.; Sala, F.; Linderman, S.; others Shrinking the generation-verification gap with weak verifiers.arXiv preprint arXiv:2506.182032025,

    arXiv

  43. [43]

    Evaluating Sample Utility for Efficient Data Selection by Mimicking Model Weights.arXiv preprint arXiv:2501.067082025,

    Huang, T.-H.; Bilkhu, M.; Cooper, J.; Sala, F.; Movellan, J. Evaluating Sample Utility for Efficient Data Selection by Mimicking Model Weights.arXiv preprint arXiv:2501.067082025,

    Pith/arXiv arXiv

  44. [44]

    Lifting weak supervision to structured prediction.Advances in Neural Information Processing Systems2022,35, 37563–37574

    Vishwakarma, H.; Sala, F. Lifting weak supervision to structured prediction.Advances in Neural Information Processing Systems2022,35, 37563–37574

  45. [45]

    E.; Kadous, M

    Ong, I.; Almahairi, A.; Wu, V .; Chiang, W.-L.; Wu, T.; Gonzalez, J. E.; Kadous, M. W.; Stoica, I. RouteLLM: Learning to Route LLMs from Preference Data. The Thirteenth International Conference on Learning Representations. 2025

    2025

  46. [46]

    J.; Bieker, J.; Li, X.; Jiang, N.; Keigwin, B.; Ranganath, G.; Keutzer, K.; Upadhyay, S

    Hu, Q. J.; Bieker, J.; Li, X.; Jiang, N.; Keigwin, B.; Ranganath, G.; Keutzer, K.; Upadhyay, S. K. RouterBench: A Benchmark for Multi-LLM Routing System. Agentic Markets Workshop at ICML 2024. 2024

    2024

  47. [47]

    Ding, D.; Mallick, A.; Wang, C.; Sim, R.; Mukherjee, S.; Rühle, V .; Lakshmanan, L. V . S.; Awadallah, A. H. Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing. The Twelfth International Conference on Learning Representations. 2024

    2024

  48. [48]

    S.; Juneja, J.; Wang, C.; Wang, Z.; Go, A.; Lee, C.-Y .; Shenoy, P.; Panigrahy, R.; Menon, A

    Jitkrittum, W.; Narasimhan, H.; Rawat, A. S.; Juneja, J.; Wang, C.; Wang, Z.; Go, A.; Lee, C.-Y .; Shenoy, P.; Panigrahy, R.; Menon, A. K.; Kumar, S. Universal Model Routing for Efficient LLM Inference. The Fourteenth International Conference on Learning Representations. 2026. 13

    2026

  49. [49]

    Large language model routing with benchmark datasets.arXiv preprint arXiv:2309.157892023,

    Shnitzer, T.; Ou, A.; Silva, M.; Soule, K.; Sun, Y .; Solomon, J.; Thompson, N.; Yurochkin, M. Large language model routing with benchmark datasets.arXiv preprint arXiv:2309.157892023,

    Pith/arXiv arXiv

  50. [50]

    N.; Thomson, M

    Hari, S. N.; Thomson, M. Tryage: Real-time, intelligent routing of user prompts to large language models.arXiv preprint arXiv:2308.116012023,

    Pith/arXiv arXiv

  51. [51]

    Routing to the expert: Efficient reward-guided ensemble of large language models

    Lu, K.; Yuan, H.; Lin, R.; Lin, J.; Yuan, Z.; Zhou, C.; Zhou, J. Routing to the expert: Efficient reward-guided ensemble of large language models. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (V olume 1: Long Papers). 2024; pp 1964–1974

    2024

  52. [52]

    RLHF Workflow: From Reward Modeling to Online RLHF.arXiv preprint arXiv:2405.078632024,

    Dong, H.; Xiong, W.; Pang, B.; Wang, H.; Zhao, H.; Zhou, Y .; Jiang, N.; Sahoo, D.; Xiong, C.; Zhang, T. RLHF Workflow: From Reward Modeling to Online RLHF.arXiv preprint arXiv:2405.078632024,

    Pith/arXiv arXiv

  53. [53]

    Anthropic Claude Opus 4.6 System Card. 2026

    2026

  54. [54]

    L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; others Gpt-4 technical report.arXiv preprint arXiv:2303.087742023,

    Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; others Gpt-4 technical report.arXiv preprint arXiv:2303.087742023,

    Pith/arXiv arXiv

  55. [55]

    Kamath, A.; Ferret, J.; Pathak, S.; Vieillard, N.; Merhej, R.; Perrin, S.; Matejovicova, T.; Ramé, A.; Rivière, M.; Rouillard, L.; others Gemma 3 technical report.arXiv preprint arXiv:2503.197862025,4

    Pith/arXiv arXiv

  56. [56]

    good” vs “bad

    Kwon, W.; Li, Z.; Zhuang, S.; Sheng, Y .; Zheng, L.; Yu, C. H.; Gonzalez, J.; Zhang, H.; Stoica, I. Efficient memory management for large language model serving with pagedattention. Proceedings of the 29th symposium on operating systems principles. 2023; pp 611–626. 14 Appendix Roadmap Our appendix is structured as follows. We begin with the prompts used ...

    2023

  57. [57]

    Evaluate how semantically relevant the response is to the question asked

    Relevance to the Query. Evaluate how semantically relevant the response is to the question asked. Measure word overlap, topic alignment, and whether the response directly addresses the core intent of the query. Penalize off-topic tangents, unrelated information, or responses that only partially address the question

  58. [58]

    Evaluate language quality and readability

    Language Quality and Readability. Evaluate language quality and readability. Check grammar correctness, spelling, punctuation, sentence variety, vocabulary richness, and overall readability. Use heuristics like average sentence length, syllable count, type-token ratio, or Flesch-like readability measures

  59. [59]

    Evaluate completeness and thoroughness of the answer

    Completeness and Coverage. Evaluate completeness and thoroughness of the answer. Check whether the response addresses all aspects and sub-questions of the query, covers edge cases, provides sufficient depth, and doesn’t leave major gaps. Penalize partial or superficial answers

  60. [60]

    Evaluate indicators of factual reliability

    Factual Accuracy Indicators. Evaluate indicators of factual reliability. Check whether the response uses language associated with verifiable facts (citations, specific names, dates, numbers), avoids hallucination red-flags (overly precise unsourced statistics, absolute claims), and shows appropriate hedging for uncertain claims. Penalize sensationalism an...

  61. [61]

    Evaluate logical coherence

    Logical Coherence and Argument Structure. Evaluate logical coherence. Check whether the response follows a clear logical flow, arguments are well-structured with premises leading to valid conclusions, transitions between ideas are smooth, and there are no internal contradictions, circular reasoning, or non-sequiturs

  62. [62]

    Evaluate clarity and conciseness

    Clarity and Conciseness. Evaluate clarity and conciseness. Score higher for responses that communicate ideas clearly and efficiently without unnecessary filler, redundant phrases, or overly convoluted sentence structures. Penalize vagueness, bloated text, and repetition of the same point in different words

  63. [63]

    Evaluate how transparently the response shows its reasoning process

    Reasoning Transparency and Step-wise Formulation. Evaluate how transparently the response shows its reasoning process. Reward responses that break down complex problems step-by-step, make intermediate conclusions visible, explain the “why” behind claims, and allow the reader to follow and verify the logic. Penalize opaque answers that jump directly to con...

  64. [64]

    likely”, “research suggests

    Epistemic Calibration and Uncertainty Communication. Evaluate how well the response communicates confidence and uncertainty. Reward responses that distinguish between well-established facts and speculative claims, use appropriate hedging language (e.g., “likely”, “research suggests”), and avoid false confidence on ambiguous topics. Penalize overconfident ...

  65. [65]

    Evaluate the structural organization of the response

    Structural Organization and Formatting. Evaluate the structural organization of the response. Reward responses that use appropriate formatting (numbered lists, bullet points, headers, paragraphs) to improve readability and information retrieval. Check for logical grouping of related ideas, clear topic sentences, and effective use of whitespace. Penalize w...

  66. [66]

    many people think

    Evidence Density and Specificity. Evaluate the density of concrete evidence and specific details in the response. Reward responses that provide specific examples, concrete data points, named entities, precise numbers, real-world references, and actionable details. Penalize vague, hand-wavy responses that use generic filler like “many people think”, “it de...

  67. [67]

    One answer is rewritten from a male-only perspective

    Gender Bias. One answer is rewritten from a male-only perspective. This measures whether the judge unfairly favors or penalizes a gender-framed answer. The calibrated program should evaluate content quality without rewarding gendered framing when it does not change the substance of the response

  68. [68]

    One answer is reformatted with markdown, headers, bullet points, and emojis, while preserving the same substantive content

    Rich-Content Bias. One answer is reformatted with markdown, headers, bullet points, and emojis, while preserving the same substantive content. This measures whether visual formatting inflates perceived quality. The calibrated program may reward organization only when it improves clarity, but must not over-reward decorative formatting, emojis, or superfici...

  69. [69]

    One answer is augmented with fake or irrelevant citations

    Reference Bias. One answer is augmented with fake or irrelevant citations. This measures whether the judge conflates the appearance of scholarly authority with actual quality. The calibrated program should not reward citations, URLs, book titles, or reference-like patterns unless they are substantively relevant to the query

  70. [70]

    One answer is padded with filler text to make it longer

    Verbosity Bias. One answer is padded with filler text to make it longer. This measures whether the judge equates length with quality. The calibrated program should reward completeness and useful detail, but penalize redundancy, filler, and length that does not add relevant information. IMPORTANT REQUIREMENTS: – Read every provided judge program before rev...

  71. [71]

    JudgeLM-100K[ 3]: 100K instruction-following response pairs annotated by GPT-4 with quality scores and rationales, originally designed for fine-tuning LLM judges

  72. [72]

    PandaLM[ 5]: Pairwise comparisons over open-source LLM outputs, with preference labels provided by GPT-3.5-Turbo

  73. [73]

    18 Table 3: Dataset statistics after filtering, along with the source of ground-truth preference labels

    MultiPref[ 27]: Real-world user prompts paired with response comparisons, annotated by both crowdworkers and domain experts. 18 Table 3: Dataset statistics after filtering, along with the source of ground-truth preference labels. Dataset Val Test Ground-Truth Source PandaLM 500 894 GPT-3.5-Turbo (val) / Human (test) MultiPref 170 1,700 Human JudgeLM 500 5...

  74. [74]

    Prometheus[ 4]: A fine-grained evaluation benchmark in which each example is paired with a scoring rubric, with feedback and preference labels generated by GPT-4

  75. [75]

    Data Filtering.To construct our evaluation set, we retain only samples with reliable preference signals and discardambiguous, tied, or low-confidence cases

    Preference-700K[ 52]: A large-scale collection of roughly 700K chosen/rejected response pairs, merged from multiple RLHF sources. Data Filtering.To construct our evaluation set, we retain only samples with reliable preference signals and discardambiguous, tied, or low-confidence cases. For human-annotated datasets (PandaLM and MultiPref), we drop samples ...

    arXiv