Graph-Structured Rubrics: Compiling Rubrics into Typed Evaluation Graphs for LLM Judges

Jie Mu; Mo Xuan; Qun Shao; Xi Chen

REVIEW 3 major objections 5 minor 23 references

Graph-Structured Rubrics: Compiling Rubrics into Typed Evaluation Graphs for LLM Judges

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Rubric rules compiled into typed graphs before judging lift LLM score agreement by up to 6.75 points.

desk verdict A genuinely new abstraction for LLM rubric composition, but its central semantic-fidelity claim is explicitly untested — the reported gains could come from arbitrary graphs rather than faithful rubric compilation. read the letter →

arxiv 2608.12097 v1 pith:CTCTMPH7 submitted 2026-08-12 cs.AI

Xi Chen , Jie Mu , Mo Xuan , Qun Shao This is my paper · ORCID

classification cs.AI

keywords LLM-as-a-judgerubriccompilationtypedevaluationgraphpointwisescoringpairwisepreferenceauditabilitycriterioncompositionDAG

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the composition rules in an evaluation rubric—how criterion-level judgments combine into a final score or preference—should be compiled into an explicit typed graph before any candidate response is seen, rather than left inside the prompt for the LLM to execute implicitly. It introduces Graph-Structured Rubrics (GSR), which compiles a rubric into a response-independent directed acyclic graph with criterion nodes, deterministic TRANSFORM, REDUCE, and GATE operator nodes, and a unique sink, then executes that graph in topological order and maps the sink to an output through a task-specific Readout. Across four pointwise datasets and two pairwise benchmarks, the paper reports that GSR under GPT-OSS-120B achieves the highest exact score agreement—0.62 to 6.75 percentage points over Prometheus-style scoring—and the highest end-to-end pairwise accuracy under native tie and abstention policies. The point is not that criterion judgments improve, but that the route they take to a final decision becomes explicit, deterministic, and replayable.

What carries the argument

The central object is the typed evaluation graph (a DAG) compiled from the rubric before candidate responses are observed: criterion nodes produce judgments, TRANSFORM/REDUCE/GATE operator nodes compose them through named, slot-ordered ports, and a unique sink feeds a task-specific Readout contract $R_\tau$. A deterministic validator rejects cycles, missing ports, arity mismatches, and type-incompatible routes; execution flows in topological order and records an audit trace that makes composition replayable. The same graph executes pointwise ($m=1$) and pairwise ($m=2$) tasks; only candidate arity, the task contract, and Readout differ.

What would settle it

Compile a rubric whose natural-language rules order a safety cap before a reduction, then inspect the accepted graph: if the first validator-approved program wires the gate to the wrong port while downstream agreement on a held-out set stays high, the assumption that structural validity suffices for faithful composition is falsified; conversely, showing that a semantic check changes the accepted graph and shifts agreement would quantify the exposure.

Watch

Extended reading notes

Core claim

The central claim is that making the composition policy of a rubric explicit as a typed, response-independent graph improves the final decision of an LLM judge without changing criterion-level judgments. GSR compiles the rubric into a program $(G, R_\tau)$ in which criterion nodes emit candidate-aligned judgments, operator nodes (TRANSFORM, REDUCE, GATE) combine them through named ports, and the unique sink produces internal quality scores that a deterministic Readout converts to a native score or a winner/tie preference. Compilation validates acyclicity, port arity, reachability, and type compatibility, accepting the first structurally valid graph without human semantic screening. The paper's experiments show that this composition layer yields higher exact agreement than direct scoring or flat weighted aggregation on the same criterion traces, and that the same graph serves both pointwise and pairwise evaluation with only candidate arity and Readout differing.

Load-bearing premise

The method's success depends on the first structurally valid graph the LLM compiler produces actually encoding the rubric's intended composition, because compilation checks only structure and types, never meaning.

Editorial extensions

If this is right

Rubric composition becomes inspectable and replayable: the audit trace records every criterion judgment, operator parameter, and slot-ordered input, so any reported score can be deterministically replayed from the recorded judgments.
A single compiled graph serves both pointwise scoring and pairwise preference, with only the Readout changing, so the same composition policy carries across task types.
Explicit graph composition shifts final ordinal boundaries even when criterion-level judgments are held fixed; the ablation reusing identical traces shows this, and the paper is explicit that GSR is a controlled composition path rather than a uniform correction of the score distribution.
The benefit is backbone-conditioned: on HelpSteer2 the exact-agreement gain over Prometheus-style is 0.25 points with one judge model and negative 1.04 points with another, so GSR controls composition but does not repair weaker criterion judgments.
In pairwise evaluation the graph interface nearly eliminates non-decisions: GSR kept 99.87--99.91% coverage with a near-zero tie/invalid rate, while checklist baselines tied 19.57--34.66% of cases.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension the paper does not run would add semantic validation of the compiled graph—for example re-deriving the natural-language rule from the graph or probing gate placement—to test whether the first structurally valid program is also the intended one.
The audit trace opens a use the paper leaves implicit: converting recorded criterion judgments into preference pairs under a declared tie policy, which could make GSR a substrate for reward-model training data rather than only a final judge.
The Readout contract's $\epsilon_\tau$ and $resolve_\tau$ are an implicit design lever: varying the tie threshold trades abstention against forced choice, and the paper's reported accuracy comparisons depend on this native policy choice.
Because cross-model results show exact-agreement gains reversing (positive with one backbone, negative with another) while Within-1 Accuracy improves in both, a direct probe of the composition layer's contribution would feed gold or human criterion-level judgments through the same graph and measure how much of the agreement gain survives.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper introduces an abstraction worth taking seriously: GSR compiles a rubric, before seeing any responses, into a typed DAG with criterion nodes, operators (TRANSFORM, REDUCE, GATE), named ports, and a task-specific Readout. That cleanly separates criterion elicitation from composition, makes the policy explicit and replayable, and is a real step beyond flat-rubric or checklist baselines. The same-trace ablation, where weighted aggregation reuses the same criterion-level judgments, is a good way to isolate the effect of graph composition. The related-work survey is broad and the distinction from DAGMetric and RULERS is plausible.

The soft spots are significant, though not fatal to the framework. The first is load-bearing: the paper states in the Compile section that it accepts the first structurally valid graph and that "whether the accepted graph captures the intended meaning of the natural-language rubric is tested through downstream agreement." Structural validation checks acyclicity, types, ports, and reachability, but it cannot detect a gate wired to the wrong criterion or a reduction that averages instead of capping. So the headline numbers are evidence about the distribution of graphs this compiler produces, not evidence that the rubric's intended policy is being executed. The stress-test note is right on this point.

Second, the margins are modest on several datasets (0.62 and 0.99 points exact agreement) and there are no significance tests or confidence intervals, just run-level standard deviations. Third, the pairwise primary metric counts ties and invalid outputs as errors, which systematically favors GSR's no-tie policy; the paper's own Valid Accuracy numbers show TICK and CheckEval doing better on valid decisions. Fourth, no code, data, or prompts are released. Fifth, cross-model results are mixed: on GLM-4.7, GSR's exact agreement is 1.04 points worse than Prometheus-style, so the benefit is backbone-dependent.

Despite all this, the paper deserves serious peer review. The abstraction is useful, the empirical work is honest — the authors explicitly acknowledge the semantic-verification gap rather than burying it — and the experiments are well-designed for isolating composition. I would send it to a strong venue, asking reviewers to focus on the semantic-fidelity issue and the tie-policy metric. A conditional accept would be defensible if the authors add a human or automated semantic check on the compiled graphs and release code. Engage with this, but don't trust the numbers until the semantic check is in.

Referee Report

3 major / 5 minor

Summary. The paper introduces Graph-Structured Rubrics (GSR), a three-stage pipeline (Compile, Execute, Readout) in which a rubric is compiled, before responses are observed, into a response-independent typed directed acyclic graph with criterion nodes, deterministic TRANSFORM/REDUCE/GATE operators, and a task-specific Readout. The graph is statically validated for structural properties; the first validator-approved program is accepted and frozen. Experiments on four pointwise datasets and two pairwise benchmarks under GPT-OSS-120B report that GSR achieves the numerically highest exact agreement and pairwise accuracy, with ablations isolating composition from criterion elicitation.

Significance. The central idea—making rubric composition an explicit, replayable program rather than implicit prompt context—is timely and well motivated. The compile/execute/readout decomposition is clean, the typed operator semantics are concrete, and the same-trace ablation is a strong design for separating graph composition from criterion elicitation. If the semantic fidelity of the compiled graph can be established, GSR would be a useful contribution to auditable LLM judging; the empirical evidence is consistent with but not yet sufficient to establish that contribution.

major comments (3)

[Compile (Eq. 5 and following paragraph)] The paper states that the first structurally valid graph is accepted and that 'whether the accepted graph captures the intended meaning of the natural-language rubric is tested through downstream agreement.' Since the validator checks only acyclicity, port/arity/type compatibility, and sink reachability, the reported agreement gains cannot distinguish faithful compilation from a compiler that systematically produces agreement-boosting graphs (for example, by smoothing scores or forcing pairwise choices). This is load-bearing for the central claim that GSR compiles rubrics into their intended evaluation policy; please add a semantic audit, such as independently annotating a sample of compiled graphs against the rubric's composition rules, or using rubrics with known compositional operators to verify compiler selection.
[Tables 1 and 2; 'Main Results'] The headline margins are small on several datasets (0.62 and 0.99 points exact agreement on SummEval Relevance and BiGGen; 0.77 and 0.28 points pairwise accuracy on MT-Bench and RubricBench), while the reported six-run standard deviations are 0.45–0.84 points (pointwise) and 0.30/0.51 points (pairwise). Without confidence intervals or significance tests, these differences are within plausible run-to-run noise, so the claim that 'GSR improves exact score agreement' is not established for those datasets. Please provide per-run results and statistical tests or confidence intervals.
[Readout, Eq. (12); Table 2] The pairwise readout depends on tie threshold ϵτ and tie/abstention resolver resolveτ, but the manuscript does not report the values used in the experiments or any sensitivity analysis. If ϵτ is tuned per benchmark, the comparison against baselines with fixed native tie policies is not apples-to-apples; if it is fixed, it should be stated. Please specify the exact tie/abstention policy used for GSR and report how Pairwise Accuracy changes with ϵτ.

minor comments (5)

[General formatting] The running header 'Graph-Structured RubricsANTGROUPRESEARCH' appears on multiple pages; please clean this up in the final version.
[Tables 1 and 3] The '+X.XX' annotations for MAE do not have a consistent sign convention; a plus sign is used for both an increase and a decrease in MAE. Add a legend or use a signed difference in the direction of improvement.
[Metrics and provenance] Standard deviations are reported only in the text; include them in the tables or an appendix for reproducibility.
[Implementation protocol] No code or artifact link is provided; since replayability is a stated contribution, a public implementation would help reviewers and users verify the claim.
[Cross-Model Sensitivity] Table 4 only includes HelpSteer2; the cross-model transfer claim would be strengthened by at least one additional dataset, ideally including a pairwise benchmark.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: GSR compiles graphs before seeing responses and never fits gold labels; the semantic-fidelity caveat is a validation gap, not a circular step.

full rationale

The paper's derivation chain is self-contained against fixed external benchmarks, so no circular reduction is present. The compiler receives no candidate responses or gold labels (Section 'Compile: From a Rubric to a Graph Program'), and the first structurally valid program is frozen and executed deterministically without any selection on agreement. Headline pointwise and pairwise results are computed against held-out reference labels (UF-TruthfulQA, HelpSteer2, SummEval Relevance, BiGGen, MT-Bench, RubricBench) rather than by fitting any parameter to those labels. The paper does explicitly concede a limitation in 'LLM-guided graph synthesis and repair': 'whether the accepted graph captures the intended meaning of the natural-language rubric is tested through downstream agreement.' This is an evidentiary gap about semantic fidelity, and using the same underlying LLM for compilation and judging is a potential bias threat, but neither step equates a prediction to its input by construction. No load-bearing self-citation, imported uniqueness theorem, or renamed known result appears; the contribution is an empirical framework evaluated against external baselines.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The framework has no fitted parameters tuned to gold labels, but it relies on several unvalidated assumptions: criterion-level LLM judgments are accurate, the first structurally valid compiled graph is semantically correct, the graph language is expressive enough, and reference labels are noise-free. The tie threshold, repair budget, and LLM-generated operator parameters are unspecified design choices that affect the reported numbers. The paper introduces methodological constructs (typed evaluation graph, Readout contract, named ports) rather than natural-world entities, so the ledger lists no invented entities.

free parameters (3)

Tie threshold epsilon_tau = Not reported
Pairwise Readout uses epsilon_tau >= 0 for winner/tie decisions; the reported 0.00% tie rate suggests a small or zero threshold, but the value is not stated.
Repair budget Krep = Not reported
Compilation retries up to Krep repairs before recording failure; the budget affects compilation success and coverage, and its value is not specified.
Operator parameters theta_omega = LLM-generated at compile time, values not reported
GATE caps, REDUCE weights, and TRANSFORM parameters are generated by the compiler and are not audited or reported, yet they determine final sink scores.

assumptions (4)

domain assumption Criterion-level judgments produced by the judge LLM are accurate enough that composition is the main source of error.
GSR's whole value proposition assumes criterion elicitation is already handled; the paper does not measure criterion-node accuracy independently.
ad hoc to paper The first structurally valid compiled graph is semantically faithful to the rubric's intended composition.
Compile accepts the first program that passes the structural validator; no semantic check is performed, and the paper admits fidelity is only tested via downstream agreement.
domain assumption The typed graph language Lambda is expressive enough to represent the composition policies of the tested rubrics.
If a rubric relies on composition not expressible in Lambda, the compiler can emit a valid-looking but wrong graph.
domain assumption Human/reference labels are treated as ground truth for agreement metrics.
No analysis of annotator noise or label disagreement is provided; reference scores are taken as correct.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Graph-Structured Rubrics: Compiling Rubrics into Typed Evaluation Graphs for LLM Judges." pith.science (2026). https://pith.science/paper/CTCTMPH7

@misc{pith2026260812097,
  author       = {Pith},
  title        = {Pith review of: Graph-Structured Rubrics: Compiling Rubrics into Typed Evaluation Graphs for LLM Judges},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CTCTMPH7}},
  note         = {Machine review of arXiv:2608.12097}
}

read the original abstract

Rubric-based evaluators commonly treat rubrics as prompt context or flat criteria: they specify what to judge but leave criterion composition implicit, even when natural-language rules state it. We introduce Graph-Structured Rubrics (GSR), which compiles a rubric into a response-independent typed evaluation graph before observing responses. Criterion nodes elicit judgments; transformation, reduction, and gating operators compose them through named ports; and a task-specific output mapping, termed Readout, converts the unique sink into a score or preference. Compilation rejects malformed or type-incompatible graphs. Pointwise evaluation judges rubric dimensions separately before graph aggregation; pairwise evaluation reuses the graph with one judgment for each candidate under every criterion. Under GPT-OSS-120B, GSR improves exact score agreement by 0.62--6.75 percentage points over Prometheus-style scoring on four pointwise datasets and achieves the numerically highest end-to-end pairwise accuracy on two preference benchmarks under native tie and abstention policies.

Figures

Figures reproduced from arXiv: 2608.12097 by the authors.

**Figure 1.** Natural-language rules leave hierarchical composition implicit within model inference; GSR routes criterion [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** GSR procedure. Compile fixes the graph before observing candidate responses; Execute runs criterion and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Six-run pointwise error geometry versus the strongest Exact Agreement baseline: pooled residuals (left) and [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Pairwise outcomes under native decision policies. Invalid outputs and ties produced by baselines are grouped [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 6 canonical work pages

[6]

Helia Hashemi, Jason Eisner, Corby Rosset, Benjamin Van Durme, and Chris Kedzie

URL https://arxiv.org/abs/2604.23581. Helia Hashemi, Jason Eisner, Corby Rosset, Benjamin Van Durme, and Chris Kedzie. LLM-rubric: A multidimensional, calibrated approach to automated evaluation of natural language texts. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 13806–13834. Association for Computati...

arXiv
[8]

Xinyu Hu, Mingqi Gao, Sen Hu, Yang Zhang, Yicheng Chen, Teng Xu, and Xiaojun Wan

URL https://arxiv.org/abs/2601.08654. Xinyu Hu, Mingqi Gao, Sen Hu, Yang Zhang, Yicheng Chen, Teng Xu, and Xiaojun Wan. Are LLM-based evaluators confusing NLG quality criteria? InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 9530–9570. Association for Computational Linguistics,

arXiv
[10]

Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo

URL https://arxiv.org/abs/2602.14069. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. Prometheus: Inducing fine-grained evaluation capability in language models. In International Conference on Learning Representations, 2024a. URL https://proceedings.iclr.c...

arXiv 2024
[12]

URL https://aclanthology.org/2025.emnlp-main.796/

doi: 10.18653/v1/2025.emnlp-main.796. URL https://aclanthology.org/2025.emnlp-main.796/. Yongqi Leng, Renren Jin, Yue Chen, Zhuowen Han, Ling Shi, Jianxiang Peng, Lei Yang, Juesi Xiao, and Deyi Xiong. Praetor: A fine-grained generative LLM evaluator with instance-level customizable evaluation criteria. InProceedings of the 63rd Annual Meeting of the Assoc...

doi:10.18653/v1/2025.emnlp-main.796 2025
[13]

URL https://aclanthology.org/2025.acl-long.513/

doi: 10.18653/v1/2025.acl-long.513. URL https://aclanthology.org/2025.acl-long.513/. Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 3214–3252. Association for Computational Linguistics,

doi:10.18653/v1/2025.acl-long.513 2025
[15]

URL https://arxiv.org/abs/2510. 07743. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-Eval: NLG evaluation using GPT-4 with better human alignment. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522. Association for Computational Linguistics,

work page 2023
[16]

URL https: //aclanthology.org/2023.emnlp-main.153/

doi: 10.18653/v1/2023.emnlp-main.153. URL https: //aclanthology.org/2023.emnlp-main.153/. Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. Calibrating LLM-based evaluator. InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluati...

doi:10.18653/v1/2023.emnlp-main.153 2023
[18]

Jon Saad-Falcon, Rajan Pathe Vivek, William Berrios, Nandita Shankar Naik, Matija Franklin, Bertie Vidgen, Amanpreet Singh, Douwe Kiela, and Shikib Mehri

URL https://arxiv.org/abs/2603.00077. Jon Saad-Falcon, Rajan Pathe Vivek, William Berrios, Nandita Shankar Naik, Matija Franklin, Bertie Vidgen, Amanpreet Singh, Douwe Kiela, and Shikib Mehri. LMUNIT: Fine-grained evaluation with natural language unit tests. InFindings of the Association for Computational Linguistics: EMNLP 2025, pages 3303–3324. Associat...

arXiv 2025

Show all 23 references

[20]

Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara

URL https://arxiv.org/abs/2602.05125. Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara. Large language models are inconsistent and biased evaluators.arXiv preprint arXiv:2405.01724,
[21]

Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis

URL https://arxiv.org/abs/2405.01724. Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. Replacing judges with juries: Evaluating LLM generations with a panel of diverse models.arXi...

arXiv
[22]

Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui

URL https://arxiv.org/abs/2404.18796. Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators.arXiv preprint arXiv:2305.17926,

arXiv
[23]

Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J

URL https://arxiv.org/abs/2305.17926. Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. HelpSteer 2: Open-source dataset for training top-performing reward models. InAdvances in Neur...

arXiv
[24]

URL https://papers.nips.cc/paper_files/paper/2024/hash/ 02fd91a387a6a5a5751e81b58a75af90-Abstract-Datasets_and_Benchmarks_Track.html

doi: 10.52202/079017-0047. URL https://papers.nips.cc/paper_files/paper/2024/hash/ 02fd91a387a6a5a5751e81b58a75af90-Abstract-Datasets_and_Benchmarks_Track.html. Tianjun Wei, Wei Wen, Ruizhi Qiao, Xing Sun, and Jianghong Ma. RocketEval: Efficient automated LLM evaluation via gr...

2024 doi
[25]

Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Min- joon Seo

URL https://proceedings.iclr.cc/paper_files/paper/ 2025/hash/937defc32e8ad2daba66a0e434177ae9-Abstract-Conference.html. Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Min- joon Seo. FLASK: Fine-grained language m...

2025
[26]

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P

URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/f41b4a6b202adcd8e150a9d4f124d8f6- Abstract-Conference.html. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gon...

2024
[27]

Junyi Zhou, Qiyuan Zhang, Yufei Wang, Fuyuan Lyu, Yidong Ming, Can Xu, Qingfeng Sun, Kai Zheng, Peng Kang, Xue Liu, and Chen Ma

URL https://arxiv.org/abs/2306.05685. Junyi Zhou, Qiyuan Zhang, Yufei Wang, Fuyuan Lyu, Yidong Ming, Can Xu, Qingfeng Sun, Kai Zheng, Peng Kang, Xue Liu, and Chen Ma. RubricBench: Aligning model-generated rubrics with human standards. InProceedings of the 64th Annual Meeting o...

arXiv
[28]

URL https://aclanthology.org/2026.acl-long.1439/

doi: 10.18653/v1/2026.acl-long.1439. URL https://aclanthology.org/2026.acl-long.1439/. Lianghui Zhu, Xinggang Wang, and Xinlong Wang. JudgeLM: Fine-tuned large language models are scalable judges. In International Conference on Learning Representations,

2026 doi
[29]

URL https://proceedings.iclr.cc/paper_files/paper/2025/hash/ 7f8f73134e253845a8f82983219a8452-Abstract-Conference.html. 11

2025
[2022]

URL https://aclanthology.org/2022.acl-long.229/

doi: 10.18653/v1/2022.acl-long.229. URL https://aclanthology.org/2022.acl-long.229/. 10 Graph-Structured RubricsANTGROUPRESEARCH Tianci Liu, Ran Xu, Tony Yu, Ilgee Hong, Carl Yang, Tuo Zhao, and Haoyu Wang. OpenRubrics: Towards scalable synthetic rubric generation for reward m...

2022
[2023]

Confident AI

URL https://arxiv.org/abs/2308.07201. Confident AI. DAG (deep acyclic graph),

arXiv
[2024]

Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun

URL https://arxiv.org/ abs/2410.03608. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. UltraFeedback: Boosting language models with scaled AI feedback. InProceedings of the 41st...

arXiv
[2025]

URL https://aclanthology.org/2025.naacl- long.303/

doi: 10.18653/v1/2025.naacl-long.303. URL https://aclanthology.org/2025.naacl- long.303/. Tae Soo Kim, Yoonjoo Lee, Jamin Shin, Young-Ho Kim, and Juho Kim. EvalLM: Interactive evaluation of large language model prompts on user-defined criteria. InProceedings of the 2024 CHI Co...

2025
[2026]

Accessed 2026-07-25

URL https://deepeval.com/docs/metrics-dag. Accessed 2026-07-25. Jonathan Cook, Tim Rocktäschel, Jakob Foerster, Dennis Aumiller, and Alex Wang. TICKing all the boxes: Generated checklists improve LLM evaluation and generation. InAdvances in Neural Information Processing Systems,

2026

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [6]

Helia Hashemi, Jason Eisner, Corby Rosset, Benjamin Van Durme, and Chris Kedzie

URL https://arxiv.org/abs/2604.23581. Helia Hashemi, Jason Eisner, Corby Rosset, Benjamin Van Durme, and Chris Kedzie. LLM-rubric: A multidimensional, calibrated approach to automated evaluation of natural language texts. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 13806–13834. Association for Computati...

arXiv

[2] [8]

Xinyu Hu, Mingqi Gao, Sen Hu, Yang Zhang, Yicheng Chen, Teng Xu, and Xiaojun Wan

URL https://arxiv.org/abs/2601.08654. Xinyu Hu, Mingqi Gao, Sen Hu, Yang Zhang, Yicheng Chen, Teng Xu, and Xiaojun Wan. Are LLM-based evaluators confusing NLG quality criteria? InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 9530–9570. Association for Computational Linguistics,

arXiv

[3] [10]

Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo

URL https://arxiv.org/abs/2602.14069. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. Prometheus: Inducing fine-grained evaluation capability in language models. In International Conference on Learning Representations, 2024a. URL https://proceedings.iclr.c...

arXiv 2024

[4] [12]

URL https://aclanthology.org/2025.emnlp-main.796/

doi: 10.18653/v1/2025.emnlp-main.796. URL https://aclanthology.org/2025.emnlp-main.796/. Yongqi Leng, Renren Jin, Yue Chen, Zhuowen Han, Ling Shi, Jianxiang Peng, Lei Yang, Juesi Xiao, and Deyi Xiong. Praetor: A fine-grained generative LLM evaluator with instance-level customizable evaluation criteria. InProceedings of the 63rd Annual Meeting of the Assoc...

doi:10.18653/v1/2025.emnlp-main.796 2025

[5] [13]

URL https://aclanthology.org/2025.acl-long.513/

doi: 10.18653/v1/2025.acl-long.513. URL https://aclanthology.org/2025.acl-long.513/. Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 3214–3252. Association for Computational Linguistics,

doi:10.18653/v1/2025.acl-long.513 2025

[6] [15]

URL https://arxiv.org/abs/2510. 07743. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-Eval: NLG evaluation using GPT-4 with better human alignment. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522. Association for Computational Linguistics,

work page 2023

[7] [16]

URL https: //aclanthology.org/2023.emnlp-main.153/

doi: 10.18653/v1/2023.emnlp-main.153. URL https: //aclanthology.org/2023.emnlp-main.153/. Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. Calibrating LLM-based evaluator. InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluati...

doi:10.18653/v1/2023.emnlp-main.153 2023

[8] [18]

Jon Saad-Falcon, Rajan Pathe Vivek, William Berrios, Nandita Shankar Naik, Matija Franklin, Bertie Vidgen, Amanpreet Singh, Douwe Kiela, and Shikib Mehri

URL https://arxiv.org/abs/2603.00077. Jon Saad-Falcon, Rajan Pathe Vivek, William Berrios, Nandita Shankar Naik, Matija Franklin, Bertie Vidgen, Amanpreet Singh, Douwe Kiela, and Shikib Mehri. LMUNIT: Fine-grained evaluation with natural language unit tests. InFindings of the Association for Computational Linguistics: EMNLP 2025, pages 3303–3324. Associat...

arXiv 2025