Pith. sign in

REVIEW 4 major objections 5 minor 76 references

Scoring rubrics for open-ended LLM outputs can be compressed to a fraction of their original count and made more reliable by selecting on measured judge agreement and item information.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 10:43 UTC pith:XGDJHRTJ

load-bearing objection The method is a sensible new combination and the paper is honestly written, but the headline numbers lean on a post-hoc threshold pick and a self-referential compression metric; worth refereeing with revisions in mind. the 4 major comments →

arxiv 2607.29252 v1 pith:XGDJHRTJ submitted 2026-07-31 cs.CL cs.AIcs.LG

CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation

Mengting Chen , Yanshu Sun , Wanting Liang , Beidi Luan , Rui Sun , Dezhi Chen , Jing Li , Zuo Bai This is my paper
classification cs.CL cs.AIcs.LG
keywords LLM evaluationrubric bankitem response theorysubmodular greedy selectionBayesian measurabilitytask-adaptive scoringrank fidelityLLM-as-a-judge
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to make open-ended LLM evaluation cheaper and more reliable by treating scoring rubrics as test items. The authors replace two heuristics in consensus-derived pipelines—strict unanimity and binary variance filtering—with a Bayesian measurability score (how consistently judges can apply a rubric) and an item-response-theory (IRT) information score (how much a rubric distinguishes systems of different ability). They then assemble compact 'rubric banks' with a submodular greedy selection over a log-information utility. The claims: measurability filtering raises human-gold agreement on JudgmentBench from κ=0.604 to 0.743; IRT-based selection improves rank fidelity over random selection in all six response blocks and reaches the target correlation on FinResearch decision-support tasks with 49 rubrics instead of 131; and corrupting task-type labels cuts the top-bottom score gap from 94.86 to 83.95 points, showing task-adaptive scoring is causally meaningful.

Core claim

CalibratedRubric claims that the two hard filters used in existing consensus-based rubric pipelines—requiring all judges to agree on a rubric and requiring its responses to vary—are crude proxies for the properties that actually matter. It replaces unanimity with a Beta–Bernoulli posterior mean over a rubric's 'measurability' (the probability that competent judges apply it consistently) and replaces variance with the item information function of 2PL IRT, integrated over the fitted ability distribution of the evaluated systems. Rubrics that pass the measurability gate are greedily assembled under a submodular log-coverage utility, with weights proportional to expected information. Empirically

What carries the argument

Beta–Bernoulli posterior measurability: a rubric's retention score E[q_j|Y]=(α+n_agree)/(α+β+n_total), using a uniform prior, replaces literal unanimity. Item information function: for binary rubrics, I_j(θ)=a_j^2 P_j(θ)(1−P_j(θ)) under a 2PL model, integrated with the fitted ability density to score each rubric. Submodular utility/greedy assembly (IIF-Greedy): U(S)=Σ_g π_g log(1+I_S(θ_g)), which is monotone submodular and yields a (1−1/e) guarantee; the greedy marginal gain discounts already-covered ability regions. Task-typing front end maps each query to one of four cognitive types (evidence reasoning, decision support, high-risk constraint, creative divergent) and selects the response sc

Load-bearing premise

The whole compression argument rests on the assumption that the full-pool IRT ability ranking is the correct target: if LLM judges' label inflation or local item dependence biases that reference, the compact bank preserves a biased ranking, and the identical human-reference correlation for adaptive and binary baselines does not yet separate the compressed bank from the baseline.

What would settle it

For any response block where IIF-Greedy is reported to beat random selection, re-estimate the reference ability vector using only human expert labels (or another high-quality gold panel) and compute the compact bank's rank correlation against that reference. If the 49-rubric bank achieves no better fidelity than random selection, or if a residual-correlation diagnostic finds large positive local dependence among retained rubrics, the information-additivity and compression claims are refuted for that block.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • On FinResearch decision-support tasks, the compact bank needs 49 rubrics rather than 131 to reach the target rank correlation; on the other blocks, greedy selection needs 43–273 items versus 142–1,226 for random selection.
  • Measurability filtering pushes human-gold agreement from κ=0.604 to 0.743 on JudgmentBench and raises inter-judge agreement monotonically as the threshold tightens, but only when at least three judges are available.
  • The IRT-selection component is the most consistent source of gain across datasets in the ablation; task typing's causal contribution is shown by the 10.91-point drop in system separation when 30% of task labels are corrupted.
  • Ranking stability: adaptive and binary scorers agree strongly on system ordering (ρ=0.8833 against the human reference on FinResearch), and the method reports capability tiers with bootstrap confidence intervals rather than fine-grained leaderboards when the data cannot support them.
  • The authors state the main benefit is a more reliable and economical evaluation instrument, not a ranking reversal; they explicitly defer several diagnostics (local item dependence, multidimensionality) to future work.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the full-pool IRT ability vector is contaminated by LLM judges' positive-label inflation (55.6–62.9% positive vs 47.1% human on JudgmentBench) or by unexamined local item dependence, then a bank that reproduces that vector preserves a possibly biased ranking; the reported human-reference correlation for FinResearch is identical for adaptive and binary baselines and does not yet validate the com
  • A direct test not in the paper: compute the compact bank's rank correlation against human-gold labels per block, rather than against the full-pool IRT estimate; if the human-gold fidelity of the compressed bank is no better than random selection's, the compression claim is mainly about reproducing LLM-judge agreement.
  • The task-type taxonomy is hand-specified (four coarse types with fixed weights such as 5× actionability); one could learn the scale-and-weight map per task cluster from data, which would test whether the typed scoring rule is genuinely optimal or merely a useful prior.
  • Because the method is generator-agnostic and cannot recover dimensions absent from the candidate pool, its ceiling is set by the initial rubric generator; combining it with adversarial or model-generated criterion expansion could extend coverage, but that would test the generator rather than the selector.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes CalibratedRubric, a pipeline that selects compact, task-adaptive rubric banks for LLM evaluation. It combines a four-type task taxonomy with hand-set scoring rules, a Beta–Bernoulli measurability filter, and IRT-based greedy rubric assembly using a submodular information-coverage utility. Theoretical propositions (unanimity attrition, reliability-aware consensus, strict refinement, submodular guarantee) are developed in Section 2, and experiments on FinResearchBench, HealthBench, HelloBench, and JudgmentBench are reported in Section 4. The three headline empirical claims are: measurability filtering raises JudgmentBench human-gold agreement from κ=0.604 to 0.743; IIF-Greedy improves cross-fitted rank-fidelity AUC over random selection in all six blocks and reaches the target correlation with 49 rather than 131 rubrics on FinResearch decision-support; and corrupting 30% of task-type labels reduces the top-bottom gap from 94.86 to 83.95 points. The paper is candid about several limitations, including unexamined local dependence, 1PL/2PL model ambiguity, and judge–human label inflation.

Significance. If the empirical claims survive closer scrutiny, this is a useful contribution to the rubric-based LLM evaluation literature. The formal propositions in Section 2, the submodular greedy guarantee, the half-split cross-fitting protocol for rank fidelity, and the bootstrap-based uncertainty reporting are concrete strengths. The proposed separation between measurability as a feasibility constraint and information as a weighting signal is clean and sensible. However, the headline κ improvement is a selected threshold maximum rather than a pre-specified estimate, and the compression claims are defined relative to an IRT-reconstructed reference, so the current evidence is suggestive rather than decisive. With a corrected evaluation protocol and tightened claims about what is actually tested, the paper would provide a solid practical method and a useful benchmark for future work.

major comments (4)
  1. [§4.2, Table 3] The headline human-gold gain (κ=0.604→0.743) is the strictest setting of a threshold sweep over θ∈{0.50,0.60,0.65,0.70,0.80}; θ directly controls the feasible set F in Eq. (15). Because τ_c is described as a dataset-level control (§3.5), the same human-gold data appear to have been used to select and to report the threshold. The reported gain is therefore a maximum over a small grid, not an unbiased estimate. Please report κ at every swept threshold or prespecify θ, and/or apply a validation split. If the selected threshold remains, state explicitly that the abstract's number is a selected maximum, not a cross-validated estimate.
  2. [§4.3, Table 4] Rank fidelity is Spearman's ρ against a full-pool/half-B ability vector estimated with the same 1PL/2PL family used for selection. The half-split refitting is a genuine strength, but the target is still an IRT-constructed reference, so a compact bank that reproduces it may simply be reproducing model-family assumptions. The external human reference (ρ=0.8833, Table 11) is identical for adaptive and binary baselines and is computed at the full pool, not at the compressed bank. Please report compressed-bank rank fidelity against the human reference where available, and/or show sensitivity to 1PL vs 2PL; the paper's own Appendix B.5 A2 (local dependence unexamined) makes this caveat necessary.
  3. [§3.4 vs §4.3] The model-selection rule in Eq. (13) chooses 1PL in five of six blocks, yet the assembler used throughout is the regularized 2PL. This is internally inconsistent for a paper that motivates careful IRT calibration. The distinction matters because discrimination parameters are weakly identified at small M, as the paper notes in A4. Please report the Table 4 AUC/compression numbers using the model selected by Eq. (13), or justify the 2PL choice explicitly; at minimum, add a sensitivity column showing how much the compression ratios change under 1PL.
  4. [§3.3, Intro contribution (3)] The claimed 'Bayesian judge-calibration procedure that models bias and reliability' is Proposition 2, but §3.3 explicitly says it 'is not instantiated in the present experiments.' The evaluated procedure is a Beta–Bernoulli posterior over all-judge agreement (Eq. 12), which has no judge-specific parameters. This gap between claimed and tested method should be fixed either by instantiating Prop 2 on the human-anchored JudgmentBench subset or by restricting the contribution claims to 'agreement-based measurability filtering.'
minor comments (5)
  1. [§4.1] The text says 'This section evaluates the task-adaptive frontend of ConsensusMultiRubric' — should read CalibratedRubric.
  2. [Table 5] The B0 row contains dashes in the IRT/Bayes columns and the 'Signal' cell is empty; the table is otherwise informative but the B0 entry should be completed or the table caption should explain the dash convention.
  3. [§4.2, Table 3] The abbreviation 'L1' is used for the posterior measurability filter but is never defined; define it (e.g., 'layer 1 / calibration') at first use.
  4. [§4.3] 'Summarized as AUC over 12 logarithmic budgets' is not defined. Please specify how the budget grid and the AUC aggregation are computed.
  5. [Table 4] The 'Hard' column header is ambiguous; the text defines it as 'random sampling within the hard-filtered set,' but the column should be labeled accordingly to avoid confusion with the 'Hard' filter itself.

Circularity Check

1 steps flagged

JudgmentBench κ gain is the best of a same-data threshold sweep; remaining derivation is self-contained.

specific steps
  1. fitted input called prediction [Abstract; §4.2, Table 3; §3.2/§2.3 (τ_c as dataset-level control)]
    "We sweep θ∈{0.50,0.60,0.65,0.70,0.80} against unfiltered majority voting. Agreement increases monotonically as the threshold becomes stricter, at the expected cost of lower coverage. From θ=0.50 to 0.80, κ rises from 0.8885 to 0.9708 for FinResearch decision support, from 0.8994 to 0.9732 for evidence reasoning, and from 0.661 to 0.743 against human gold on JudgmentBench. Table 3 reports the strictest setting."

    The abstract's headline improvement (κ=0.604→0.743) is the value at θ=0.80, which is the strictest of five thresholds swept on the same JudgmentBench human-gold data. Because the paper reports that agreement increases monotonically with θ, 'the strictest setting' is the maximum of the sweep. τ_c is not pre-specified: §2.3 says 'the budget B and the validity threshold τ_c remain the only quantities set per dataset,' and §3.2 calls them 'dataset-level controls.' Thus the measured improvement is an in-sample selected maximum, not an out-of-sample or pre-registered estimate; the claim 'filtering improves agreement from 0.604 to 0.743' is partly manufactured by choosing the threshold that maximizes it on the evaluation set.

full rationale

The only load-bearing reduction I can exhibit is the JudgmentBench threshold sweep: the headline κ value is selected as the best of five θ values evaluated on the same human-gold data, so the magnitude of the improvement is by construction the maximum over the sweep. The other main claims are not circular in the same way. The IRT rank-fidelity comparison is cross-fitted: 'selection and refitting use half A, while the reference ability vector is estimated from half B,' with 20 splits, so the target ranking is out-of-sample even though it shares the same 1PL/2PL family. The paper also provides external anchors (human-gold agreement, human reference ρ=0.8833) that are not generated by the method. The baseline from Luan et al. (2026) is a self-citation, but it is used as a comparator and for one illustrative attrition extrapolation, not as the justification of the central derivation; the submodular guarantee is cited to Nemhauser et al. (1978). I therefore score 6 rather than higher: one central quantitative claim reduces to a same-data selection, but the rest of the empirical chain has genuine independent content.

Axiom & Free-Parameter Ledger

5 free parameters · 8 axioms · 0 invented entities

The central results depend on hand-set thresholds and weights, on untested IRT assumptions (unidimensionality, local independence, judge conditional independence), and on a reference ranking generated by the same model family being optimized. The candidate-pool completeness assumption is acknowledged by the authors.

free parameters (5)
  • Measurability threshold τ_c (θ) = 0.80 (strictest of swept {0.50,0.60,0.65,0.70,0.80})
    Retains rubrics with posterior agreement mean ≥ θ; the headline κ=0.743 uses the strictest setting, so the threshold is selected post hoc (§3.3, §4.2).
  • Budget B = 28/49 for FinResearch blocks; 43–273 across blocks
    Cardinality constraint in Eq. 2; set per dataset and via target-correlation threshold (§4.3).
  • Task-type scoring weights = 5× actionability, 3× creativity, -0.3/-0.5 risk penalties
    Hand-set in Table 1/8; no derivation, learning, or ablation is provided (§2.4).
  • Prior scale for log a_j = 0.5
    log a_j ~ N(0, 0.5^2) chosen to prevent separating items from dominating; not estimated (§3.4).
  • Target minimum-bank correlation = .95 (or .9429 for six systems)
    Defines 'reaching the target correlation'; arbitrary threshold that determines the 49-vs-131 headline (§4.3).
axioms (8)
  • domain assumption A1: dominant latent dimension
    2PL/1PL assumes one dominant capability; residual-spectrum checks are only partially reported (Appendix B.5).
  • domain assumption A2: local item independence
    Rubrics from the same output may be dependent given θ; Yen's Q3 and dependent-item refitting are not completed, so the additive test information and submodular guarantee are conditional (Appendix B.5).
  • domain assumption A3: conditional independence of judges
    LLM judges share training data and conventions; correlated errors can inflate apparent consensus (Appendix B.5).
  • domain assumption A4: small-M identifiability
    Joint 2PL estimation is poorly conditioned with ~10 systems; priors and 1PL comparison are used instead (Appendix B.5).
  • domain assumption Judge error model (Assumption 1)
    Judges label independently with per-judge error rates; used for Proposition 1-2's attrition and Bayes-optimal consensus (Section 2.2).
  • domain assumption Candidate pools are complete
    The method cannot recover dimensions absent from C; if the generator misses a quality dimension the optimum bank is incomplete (Section 3).
  • domain assumption Agreement posterior measures rubric measurability
    q_j is estimated purely from inter-judge agreement with no human labels; the paper itself states this is necessary but not sufficient for expert endorsement (Sections 2.1, 3.3).
  • ad hoc to paper Task-type classifier accuracy
    §3.1: a prompted classifier assigns each query to a task type; no validation accuracy or calibration is reported, and the scoring rules depend on its output.

pith-pipeline@v1.3.0-daily-deepseek · 14071 in / 16570 out tokens · 136332 ms · 2026-08-03T10:43:32.309195+00:00 · methodology

0 comments
read the original abstract

Reliable evaluation of open-ended LLM outputs requires fine-grained rubrics, yet expert curation is costly and difficult to scale. Existing automated pipelines rely on strict judge unanimity and binary variance filters, which cannot distinguish measurable rubrics from informative ones. We introduce CalibratedRubric, a task-adaptive framework that combines type-specific scoring, Bayesian rubric-measurability filtering, and item response theory (IRT)-based bank assembly. CalibratedRubric estimates each rubric's measurability with a Beta--Bernoulli agreement posterior and uses a submodular information-coverage objective to construct compact rubric banks over the observed capability range. Across financial, healthcare, general, and legal benchmarks, measurability filtering improves human-gold agreement on JudgmentBench from $\kappa=0.604$ to $0.743$. IRT-based greedy selection improves cross-fitted rank fidelity over random selection across all six evaluated response blocks and requires only 49 rather than 131 rubrics to reach the target correlation on FinResearchBench decision-support tasks. Task-label perturbations further reduce system separation, confirming the practical relevance of task-adaptive scoring. These results support CalibratedRubric as an efficient, uncertainty-aware approach to open-ended LLM evaluation, with calibration gains depending on sufficient judge redundancy.

Figures

Figures reproduced from arXiv: 2607.29252 by Beidi Luan, Dezhi Chen, Jing Li, Mengting Chen, Rui Sun, Wanting Liang, Yanshu Sun, Zuo Bai.

Figure 1
Figure 1. Figure 1: CalibratedRubric. Task typing determines the scoring rule; Bayesian rubric calibration supplies the measurability [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

76 extracted references · 8 canonical work pages · 3 internal anchors

  1. [1]

    K.; Wei, J.; Hicks, R

    Arora, R. K.; Wei, J.; Hicks, R. S.; Bowman, P.; Qui \ n onero-Candela, J.; Tsimpourlas, F.; Sharman, M.; Shah, M.; Vallone, A.; Beutel, A.; Heidecke, J.; and Singhal, K. 2025. HealthBench : Evaluating Large Language Models Towards Improved Human Health. arXiv:2505.08775

    Pith/arXiv arXiv 2025

  2. [2]

    Baker, F. B. 2001. The Basics of Item Response Theory. ERIC Clearinghouse on Assessment and Evaluation, 2 edition. ISBN 1-886047-03-0

    2001

  3. [3]

    Birnbaum, A. 1968. Some Latent Trait Models and Their Use in Inferring an Examinee's Ability. In Lord, F. M.; and Novick, M. R., eds., Statistical Theories of Mental Test Scores, 397--479. Addison-Wesley

    1968

  4. [4]

    D.; and Aitkin, M

    Bock, R. D.; and Aitkin, M. 1981. Marginal Maximum Likelihood Estimation of Item Parameters: Application of an EM Algorithm. Psychometrika, 46(4): 443--459

    1981

  5. [5]

    J.; Ashktorab, Z.; Geyer, W.; Li, T.; and Gomez-Zara, D

    Chiang, C.; Gebreegziabher, S.; Szymanski, A.; Yang, Y.; Do, H. J.; Ashktorab, Z.; Geyer, W.; Li, T.; and Gomez-Zara, D. 2026. MultEval : Supporting Collaborative Alignment for LLM -as-a-Judge Evaluation Criteria. In Proceedings of the 5th Annual Symposium on Human-Computer Interaction for Work, 1--17. Association for Computing Machinery

    2026

  6. [6]

    Cook, J.; Rockt \"a schel, T.; Foerster, J.; Aumiller, D.; and Wang, A. 2024. TICK ing All the Boxes: Generated Checklists Improve LLM Evaluation and Generation. arXiv:2410.03608

    Pith/arXiv arXiv 2024

  7. [7]

    P.; and Skene, A

    Dawid, A. P.; and Skene, A. M. 1979. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), 28(1): 20--28

    1979

  8. [8]

    Hashemi, H.; Eisner, J.; Rosset, C.; Van Durme, B.; and Kedzie, C. 2024. LLM -Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 13806--13834. Association for Computational Linguistics

    2024

  9. [9]

    Jia, M.; Zhang, Z.; Cases, I.; Liu, Z.; Jiang, M.; and Qi, P. 2026. AutoRubric : Rubric-Based Generative Rewards for Faithful Multimodal Reasoning. In Findings of the Association for Computational Linguistics: ACL 2026, 25707--25724. Association for Computational Linguistics

    2026

  10. [10]

    Kim, S.; Shin, J.; Cho, Y.; Jang, J.; Longpre, S.; Lee, H.; Yun, S.; Shin, S.; Kim, S.; Thorne, J.; and Seo, M. 2024. Prometheus: Inducing Fine-Grained Evaluation Capability in Language Models. In The Twelfth International Conference on Learning Representations

    2024

  11. [11]

    P.; Wu, H.; and Yu, H

    Lalor, J. P.; Wu, H.; and Yu, H. 2016. Building an Evaluation Scale using Item Response Theory. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 648--657. Association for Computational Linguistics

    2016

  12. [12]

    Li, H.; Chen, J.; Ai, Q.; Chu, Z.; Zhou, Y.; Dong, Q.; and Liu, Y. 2025. CalibraEval : Calibrating Prediction Distribution to Mitigate Selection Bias in LLM s-as-Judges. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 16537--16552. Association for Computational Linguistics

    2025

  13. [13]

    Li, J.; Sun, S.; Yuan, W.; Fan, R.-Z.; Zhao, H.; and Liu, P. 2024. Generative Judge for Evaluating Alignment. In The Twelfth International Conference on Learning Representations

    2024

  14. [14]

    Li, S.; Zhao, J.; Ren, H.; Wei, Z.; Zhou, Y.; Yang, J.; Liu, S.; Zhang, K.; and Wei, C. 2026. RubricHub : A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generation. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 31320--31344. Association for Computa...

    2026

  15. [15]

    Lin, C.-Y. 2004. ROUGE : A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, 74--81. Association for Computational Linguistics

    2004

  16. [16]

    Liu, X.; Ma, X.; Ma, Y.; Peng, Y.; Wang, D.; Wen, Z.; Zhang, G.; Zhang, K.; Chen, X.; Ding, Y.; et al. 2026. XpertBench : Expert Level Tasks with Rubrics-Based Evaluation. arXiv:2604.02368

    Pith/arXiv arXiv 2026

  17. [17]

    Liu, Y.; Iter, D.; Xu, Y.; Wang, S.; Xu, R.; and Zhu, C. 2023. G -Eval: NLG Evaluation Using GPT -4 with Better Human Alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2511--2522. Association for Computational Linguistics

    2023

  18. [18]

    Lord, F. M. 1980. Applications of Item Response Theory to Practical Testing Problems. Lawrence Erlbaum Associates. ISBN 0-89859-006-X

    1980

  19. [19]

    Luan, B.; Sun, R.; Wang, S.; Gu, Y.; Li, C.; Xiong, Z.; Li, J.; and Bai, Z. 2026. FinResearchBench II : A Deep Research Benchmark with Consensus-Derived Gold Rubrics for Distinguishing Financial Report Quality. arXiv:2607.12252

    Pith/arXiv arXiv 2026

  20. [20]

    Maia Polo, F.; Weber, L.; Choshen, L.; Sun, Y.; Xu, G.; and Yurochkin, M. 2024. tinyBenchmarks: Evaluating LLM s with Fewer Examples. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, 34303--34326. PMLR

    2024

  21. [21]

    L.; Wolsey, L

    Nemhauser, G. L.; Wolsey, L. A.; and Fisher, M. L. 1978. An Analysis of Approximations for Maximizing Submodular Set Functions---I. Mathematical Programming, 14(1): 265--294

    1978

  22. [22]

    Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. BLEU : A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, 311--318. Association for Computational Linguistics

    2002

  23. [23]

    M.; Yang, J.; Zhang, G.; Peng, J.; Zhang, Z.; Zhang, S.; and Chen, K

    Que, H.; Duan, F.; He, L.; Mou, Y.; Zhou, W.; Liu, J.; Rong, W.; Wang, Z. M.; Yang, J.; Zhang, G.; Peng, J.; Zhang, Z.; Zhang, S.; and Chen, K. 2024. HelloBench : Evaluating Long Text Generation Capabilities of Large Language Models. arXiv:2409.16191

    Pith/arXiv arXiv 2024

  24. [24]

    M.; Lalor, J

    Rodriguez, P.; Barrow, J.; Hoyle, A. M.; Lalor, J. P.; Jia, R.; and Boyd-Graber, J. 2021. Evaluation Examples Are Not Equally Informative: How Should That Change NLP Leaderboards? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: ...

    2021

  25. [25]

    Samejima, F. 1969. Estimation of Latent Ability Using a Response Pattern of Graded Scores. Psychometrika, 34(S1): 1--97

    1969

  26. [26]

    J.; and Glas, C

    van der Linden, W. J.; and Glas, C. A. W., eds. 2000. Computerized Adaptive Testing: Theory and Practice. Kluwer Academic Publishers

    2000

  27. [27]

    M.; Huang, W.; Mungra, D.; Pang, R

    Vania, C.; Htut, P. M.; Huang, W.; Mungra, D.; Pang, R. Y.; Phang, J.; Liu, H.; Cho, K.; and Bowman, S. R. 2021. Comparing Test Sets with Item Response Theory. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 1141--...

    2021

  28. [28]

    Wang, L.; Ma, C.; Feng, X.; Zhang, Z.; Yang, H.; Zhang, J.; Chen, Z.; Tang, J.; Chen, X.; Lin, Y.; et al. 2024 a . A Survey on Large Language Model Based Autonomous Agents. Frontiers of Computer Science, 18(6): 186345

    2024

  29. [29]

    Wang, P.; Li, L.; Chen, L.; Cai, Z.; Zhu, D.; Lin, B.; Cao, Y.; Kong, L.; Liu, Q.; Liu, T.; and Sui, Z. 2024 b . Large Language Models Are Not Fair Evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 9440--9450. Association for Computational Linguistics

    2024

  30. [30]

    Wang, Z.; and Blanco, E. 2026. Generating and Refining Dynamic Evaluation Rubrics for LLM -as-a-Judge. arXiv:2605.30568

    Pith/arXiv arXiv 2026

  31. [31]

    Xi, Z.; Chen, W.; Guo, X.; He, W.; Ding, Y.; Hong, B.; Zhang, M.; Wang, J.; Jin, S.; Zhou, E.; et al. 2025. The Rise and Potential of Large Language Model Based Agents: A Survey. Science China Information Sciences, 68(2): 121101

    2025

  32. [32]

    Yang, R.; Chen, R.; Kelaita, P.; Ranjan, R.; Ma, S.; Dickens, C.; Guillod, M.; Ma, M.; and Nyarko, J. 2026. JudgmentBench : Comparing Rubric and Preference Evaluation for Quality Assessment. arXiv:2605.25240

    Pith/arXiv arXiv 2026

  33. [33]

    Ye, S.; Kim, D.; Kim, S.; Hwang, H.; Kim, S.; Jo, Y.; Thorne, J.; Kim, J.; and Seo, M. 2024. FLASK : Fine-Grained Language Model Evaluation Based on Alignment Skill Sets. In The Twelfth International Conference on Learning Representations

    2024

  34. [34]

    Yen, W. M. 1984. Effects of Local Item Dependence on the Fit and Equating Performance of the Three-Parameter Logistic Model. Applied Psychological Measurement, 8(2): 125--145

    1984

  35. [35]

    Q.; and Artzi, Y

    Zhang, T.; Kishore, V.; Wu, F.; Weinberger, K. Q.; and Artzi, Y. 2020. BERTScore : Evaluating Text Generation with BERT . In International Conference on Learning Representations

    2020

  36. [36]

    Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E.; et al. 2023. Judging LLM -as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems, volume 36, 46595--46623

    2023

  37. [37]

    2026 , publisher =

    Chiang, Charles and Gebreegziabher, Simret and Szymanski, Annalisa and Yang, Yukun and Do, Hyo Jin and Ashktorab, Zahra and Geyer, Werner and Li, Toby and Gomez-Zara, Diego , booktitle =. 2026 , publisher =. doi:10.1145/3808045.3808093 , url =

    arXiv 2026

  38. [38]

    2024 , eprint =

    Cook, Jonathan and Rockt. 2024 , eprint =. doi:10.48550/arXiv.2410.03608 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.2410.03608 2024

  39. [39]

    Findings of the Association for Computational Linguistics: ACL 2024 , pages =

    Ranking Large Language Models without Ground Truth , author =. Findings of the Association for Computational Linguistics: ACL 2024 , pages =. 2024 , publisher =. doi:10.18653/v1/2024.findings-acl.143 , url =

    work page doi:10.18653/v1/2024.findings-acl.143 2024

  40. [40]

    2024 , publisher =

    Hashemi, Helia and Eisner, Jason and Rosset, Corby and Van Durme, Benjamin and Kedzie, Chris , booktitle =. 2024 , publisher =. doi:10.18653/v1/2024.acl-long.745 , url =

    doi:10.18653/v1/2024.acl-long.745 2024

  41. [41]

    2026 , publisher =

    Jia, Mengzhao and Zhang, Zhihan and Cases, Ignacio and Liu, Zheyuan and Jiang, Meng and Qi, Peng , booktitle =. 2026 , publisher =. doi:10.18653/v1/2026.findings-acl.1282 , url =

    work page doi:10.18653/v1/2026.findings-acl.1282 2026

  42. [42]

    The Twelfth International Conference on Learning Representations , year =

    Prometheus: Inducing Fine-Grained Evaluation Capability in Language Models , author =. The Twelfth International Conference on Learning Representations , year =

  43. [43]

    Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages =

    Building an Evaluation Scale using Item Response Theory , author =. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages =. 2016 , publisher =. doi:10.18653/v1/D16-1062 , url =

    doi:10.18653/v1/d16-1062 2016

  44. [44]

    The Twelfth International Conference on Learning Representations , year =

    Generative Judge for Evaluating Alignment , author =. The Twelfth International Conference on Learning Representations , year =

  45. [45]

    2026 , publisher =

    Li, Sunzhu and Zhao, Jiale and Ren, Huimin and Wei, Zhenlin and Zhou, Yang and Yang, Jingwen and Liu, Shunyu and Zhang, Kaike and Wei, Chen , booktitle =. 2026 , publisher =. doi:10.18653/v1/2026.acl-long.1445 , url =

    work page doi:10.18653/v1/2026.acl-long.1445 2026

  46. [46]

    2004 , publisher =

    Lin, Chin-Yew , booktitle =. 2004 , publisher =

    2004

  47. [47]

    doi:10.48550/arXiv.2604.02368 , url =

    Liu, Xue and Ma, Xin and Ma, Yuxin and Peng, Yongchang and Wang, Duo and Wen, Zhoufutu and Zhang, Ge and Zhang, Kaiyuan and Chen, Xinyu and Ding, Yida and others , year =. doi:10.48550/arXiv.2604.02368 , url =. 2604.02368 , archivePrefix =

    Pith/arXiv arXiv doi:10.48550/arxiv.2604.02368

  48. [48]

    FinResearchBench II: A Deep Research Benchmark with Consensus-Derived Gold Rubrics for Distinguishing Financial Report Quality

    Luan, Beidi and Sun, Rui and Wang, Sinuo and Gu, Yan and Li, Chao and Xiong, Zhenliang and Li, Jing and Bai, Zuo , year =. doi:10.48550/arXiv.2607.12252 , url =. 2607.12252 , archivePrefix =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2607.12252

  49. [49]

    tinyBenchmarks: Evaluating

    Maia Polo, Felipe and Weber, Lucas and Choshen, Leshem and Sun, Yuekai and Xu, Gongjun and Yurochkin, Mikhail , booktitle =. tinyBenchmarks: Evaluating. 2024 , publisher =

    2024

  50. [50]

    and Jia, Robin and Boyd-Graber, Jordan , booktitle =

    Rodriguez, Pedro and Barrow, Joe and Hoyle, Alexander Miserlis and Lalor, John P. and Jia, Robin and Boyd-Graber, Jordan , booktitle =. Evaluation Examples Are Not Equally Informative: How Should That Change. 2021 , publisher =. doi:10.18653/v1/2021.acl-long.346 , url =

    doi:10.18653/v1/2021.acl-long.346 2021

  51. [51]

    Comparing Test Sets with Item Response Theory , author =. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages =. 2021 , publisher =. doi:10.18653/v1/2021.acl-long.92 , url =

    doi:10.18653/v1/2021.acl-long.92 2021

  52. [52]

    Frontiers of Computer Science , volume =

    A Survey on Large Language Model Based Autonomous Agents , author =. Frontiers of Computer Science , volume =. 2024 , doi =

    2024

  53. [53]

    Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judge

    Wang, Zijie and Blanco, Eduardo , year =. Generating and Refining Dynamic Evaluation Rubrics for. doi:10.48550/arXiv.2605.30568 , url =. 2605.30568 , archivePrefix =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2605.30568

  54. [54]

    Science China Information Sciences , volume =

    The Rise and Potential of Large Language Model Based Agents: A Survey , author =. Science China Information Sciences , volume =. 2025 , doi =

    2025

  55. [55]

    2024 , url =

    Ye, Seonghyeon and Kim, Doyoung and Kim, Sungdong and Hwang, Hyeonbin and Kim, Seungone and Jo, Yongrae and Thorne, James and Kim, Juho and Seo, Minjoon , booktitle =. 2024 , url =

    2024

  56. [56]

    and Artzi, Yoav , booktitle =

    Zhang, Tianyi and Kishore, Varsha and Wu, Felix and Weinberger, Kilian Q. and Artzi, Yoav , booktitle =. 2020 , url =

    2020

  57. [57]

    2001 , publisher =

    The Basics of Item Response Theory , author =. 2001 , publisher =

    2001

  58. [58]

    1980 , publisher =

    Applications of Item Response Theory to Practical Testing Problems , author =. 1980 , publisher =

    1980

  59. [59]

    Statistical Theories of Mental Test Scores , pages =

    Some Latent Trait Models and Their Use in Inferring an Examinee's Ability , author =. Statistical Theories of Mental Test Scores , pages =. 1968 , publisher =

    1968

  60. [60]

    Psychometrika , volume =

    Estimation of Latent Ability Using a Response Pattern of Graded Scores , author =. Psychometrika , volume =. 1969 , doi =

    1969

  61. [61]

    Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric and others , booktitle =. Judging. 2023 , url =

    2023

  62. [62]

    2000 , publisher =

    Computerized Adaptive Testing: Theory and Practice , editor =. 2000 , publisher =. doi:10.1007/0-306-47531-6 , url =

    work page doi:10.1007/0-306-47531-6 2000

  63. [63]

    , journal =

    Dawid, Alexander Philip and Skene, Allan M. , journal =. Maximum Likelihood Estimation of Observer Error-Rates Using the. 1979 , doi =

    1979

  64. [64]

    Applied Psychological Measurement , volume =

    Effects of Local Item Dependence on the Fit and Equating Performance of the Three-Parameter Logistic Model , author =. Applied Psychological Measurement , volume =. 1984 , doi =

    1984

  65. [65]

    2002 , publisher =

    Papineni, Kishore and Roukos, Salim and Ward, Todd and Zhu, Wei-Jing , booktitle =. 2002 , publisher =. doi:10.3115/1073083.1073135 , url =

    arXiv 2002

  66. [66]

    Darrell and Aitkin, Murray , journal =

    Bock, R. Darrell and Aitkin, Murray , journal =. Marginal Maximum Likelihood Estimation of Item Parameters: Application of an. 1981 , doi =

    1981

  67. [67]

    2011 , eprint =

    Bayesian Active Learning for Classification and Preference Learning , author =. 2011 , eprint =. doi:10.48550/arXiv.1112.5745 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.1112.5745 2011

  68. [68]

    and Wei, Jason and Hicks, Rebecca Soskin and Bowman, Preston and Qui

    Arora, Rahul K. and Wei, Jason and Hicks, Rebecca Soskin and Bowman, Preston and Qui. 2025 , eprint =. doi:10.48550/arXiv.2505.08775 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.08775 2025

  69. [69]

    doi:10.48550/arXiv.2409.16191 , url =

    Que, Haoran and Duan, Feiyu and He, Liqun and Mou, Yutao and Zhou, Wangchunshu and Liu, Jiaheng and Rong, Wenge and Wang, Zekun Moore and Yang, Jian and Zhang, Ge and Peng, Junran and Zhang, Zhaoxiang and Zhang, Songyang and Chen, Kai , year =. doi:10.48550/arXiv.2409.16191 , url =. 2409.16191 , archivePrefix =

    Pith/arXiv arXiv doi:10.48550/arxiv.2409.16191

  70. [70]

    JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment

    Yang, Russell and Chen, Ruishi and Kelaita, Pierce and Ranjan, Riya and Ma, Sibo and Dickens, Charles and Guillod, Matthew and Ma, Megan and Nyarko, Julian , year =. doi:10.48550/arXiv.2605.25240 , url =. 2605.25240 , archivePrefix =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2605.25240

  71. [71]

    Mathematical Programming , volume =

    An Analysis of Approximations for Maximizing Submodular Set Functions---I , author =. Mathematical Programming , volume =. 1978 , doi =

    1978

  72. [72]

    Advances in Neural Information Processing Systems , volume =

    Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise , author =. Advances in Neural Information Processing Systems , volume =. 2009 , url =

    2009

  73. [73]

    2023 , publisher =

    Liu, Yang and Iter, Dan and Xu, Yichong and Wang, Shuohang and Xu, Ruochen and Zhu, Chenguang , booktitle =. 2023 , publisher =. doi:10.18653/v1/2023.emnlp-main.153 , url =

    doi:10.18653/v1/2023.emnlp-main.153 2023

  74. [74]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    Large Language Models Are Not Fair Evaluators , author =. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =. 2024 , publisher =. doi:10.18653/v1/2024.acl-long.511 , url =

    doi:10.18653/v1/2024.acl-long.511 2024

  75. [75]

    2025 , publisher =

    Li, Haitao and Chen, Junjie and Ai, Qingyao and Chu, Zhumin and Zhou, Yujia and Dong, Qian and Liu, Yiqun , booktitle =. 2025 , publisher =. doi:10.18653/v1/2025.acl-long.808 , url =

    work page doi:10.18653/v1/2025.acl-long.808 2025

  76. [76]

    Educational and Psychological Measurement , volume =

    A Coefficient of Agreement for Nominal Scales , author =. Educational and Psychological Measurement , volume =. 1960 , doi =

    1960