{"id":"96fd60f6-6246-4004-9a17-9936cc3f9256","arxiv_id":"2608.12097","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Compiling rubrics into typed evaluation graphs before seeing responses improves LLM judge agreement on four pointwise and two pairwise benchmarks over Prometheus-style and checklist baselines.","lead":"Graph-Structured Rubrics turns a written rubric into a fixed typed graph, so an LLM judge scores or compares responses by following the graph instead of reinterpreting the rubric each time. On six benchmarks the method beats standard LLM judges on exact score agreement and pairwise preference accuracy, though margins are small and code is not released.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Explicit limitation: the first validator-approved graph is accepted with no semantic check, so downstream agreement cannot attribute the reported gains to faithful rubric compilation; the method's central claim is therefore unverified.","rationale":"The reader's CONDITIONAL verdict is appropriate, and my concern coincides with their weakest_assumption. The strongest_claim is empirical: agreement gains under GPT-OSS-120B. Those numbers could in principle survive even if compiled graphs are semantically arbitrary. But the paper's own contribution is defined as compiling a rubric's composition policy into an executable, response-independent graph; if the accepted graph does not match the rubric's intended meaning, the method is not doing what it claims, and the measured gains are uninterpretable as evidence for rubric-faithful composition. The Compile section is explicit about this: no semantic screening is performed, and semantic fidelity is tested only through downstream agreement. Downstream agreement is insufficient as a validity check because a graph can improve agreement for reasons unrelated to rubric semantics, such as correcting judge bias, imposing score smoothing, or eliminating abstentions in pairwise tasks. The same-trace ablation is good evidence that graph composition changes decision boundaries relative to flat weighting, but it does not establish that the composition is the rubric's composition. Therefore the load-bearing assumption is semantic fidelity of the first-valid compiled graph, exactly as the reader states. The proposed audit—human comparison of accepted graphs to rubric text, plus correlation of fidelity with per-instance gain—would settle whether the concern lands. If fidelity is high and gains track fidelity, the concern is resolved; if not, the empirical headline cannot be attributed to rubric compilation and the central claim would need to be weakened. Pairwise tie-policy issues and lack of significance testing are real but secondary: they affect interpretation of particular table rows, while semantic fidelity affects the identity of the method itself. No verdict change from the reader's CONDITIONAL is needed unless the audit fails.","tokens_in":12909,"tokens_out":6761,"duration_ms":59780,"concrete_test":"Sample 100 accepted graphs per dataset (400 total). For each, give an independent annotator (or rubric author, blinded to GSR's predictions) the rubric text and the compiled graph (nodes, named ports, operators, parameters, and Readout), and ask whether the graph faithfully encodes the rubric's composition rules: order of reductions, gates/caps, and tie or forced-choice policy. Compute the semantic-fidelity rate. Then split the GSR-versus-baseline exact-agreement gain by fidelity: if gains concentrate on faithful graphs, rubric compilation is the mechanism; if gains appear equally on unfaithful graphs, the headline effect is not attributable to rubric semantics. This single audit would settle whether the central claim holds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The Compile section (LLM-guided graph synthesis and repair) states that the first structurally valid program is accepted, 'no manual semantic screening or selection among valid candidates is performed,' and that 'whether the accepted graph captures the intended meaning of the natural-language rubric is tested through downstream agreement.' This is the load-bearing point: all headline numbers come from graphs that are never checked against the rubric's intended composition. Structural validation only enforces acyclicity, port/arity/type compatibility, and sink reachability; it cannot detect, for example, a GATE wired to the wrong criterion, a REDUCE that averages instead of applying a rubric-specified cap, or a Readout that resolves near-ties in the wrong direction. If the compiler has a systematic bias toward graphs that happen to improve agreement (by smoothing scores, correcting judge biases, or forcing pairwise choices), the reported gains are evidence about the compiler's graph distribution, not about faithful execution of the rubric. Because downstream agreement is the only test, the central conceptual contribution—compiling rubrics into their intended evaluation policy—is not actually verified. The same-trace ablation shows that graph composition can change decision boundaries, but it does not show that the composition corresponds to the rubric. The pairwise tie-policy issue is real but secondary; the primary gap is semantic.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Graph-Structured Rubrics (GSR), a three-stage pipeline (Compile, Execute, Readout) in which a rubric is compiled, before responses are observed, into a response-independent typed directed acyclic graph with criterion nodes, deterministic TRANSFORM/REDUCE/GATE operators, and a task-specific Readout. The graph is statically validated for structural properties; the first validator-approved program is accepted and frozen. Experiments on four pointwise datasets and two pairwise benchmarks under GPT-OSS-120B report that GSR achieves the numerically highest exact agreement and pairwise accuracy, with ablations isolating composition from criterion elicitation.","tokens_in":13158,"tokens_out":5281,"duration_ms":45966,"significance":"The central idea—making rubric composition an explicit, replayable program rather than implicit prompt context—is timely and well motivated. The compile/execute/readout decomposition is clean, the typed operator semantics are concrete, and the same-trace ablation is a strong design for separating graph composition from criterion elicitation. If the semantic fidelity of the compiled graph can be established, GSR would be a useful contribution to auditable LLM judging; the empirical evidence is consistent with but not yet sufficient to establish that contribution.","major_comments":[{"comment":"The paper states that the first structurally valid graph is accepted and that 'whether the accepted graph captures the intended meaning of the natural-language rubric is tested through downstream agreement.' Since the validator checks only acyclicity, port/arity/type compatibility, and sink reachability, the reported agreement gains cannot distinguish faithful compilation from a compiler that systematically produces agreement-boosting graphs (for example, by smoothing scores or forcing pairwise choices). This is load-bearing for the central claim that GSR compiles rubrics into their intended evaluation policy; please add a semantic audit, such as independently annotating a sample of compiled graphs against the rubric's composition rules, or using rubrics with known compositional operators to verify compiler selection.","section":"Compile (Eq. 5 and following paragraph)"},{"comment":"The headline margins are small on several datasets (0.62 and 0.99 points exact agreement on SummEval Relevance and BiGGen; 0.77 and 0.28 points pairwise accuracy on MT-Bench and RubricBench), while the reported six-run standard deviations are 0.45–0.84 points (pointwise) and 0.30/0.51 points (pairwise). Without confidence intervals or significance tests, these differences are within plausible run-to-run noise, so the claim that 'GSR improves exact score agreement' is not established for those datasets. Please provide per-run results and statistical tests or confidence intervals.","section":"Tables 1 and 2; 'Main Results'"},{"comment":"The pairwise readout depends on tie threshold ϵτ and tie/abstention resolver resolveτ, but the manuscript does not report the values used in the experiments or any sensitivity analysis. If ϵτ is tuned per benchmark, the comparison against baselines with fixed native tie policies is not apples-to-apples; if it is fixed, it should be stated. Please specify the exact tie/abstention policy used for GSR and report how Pairwise Accuracy changes with ϵτ.","section":"Readout, Eq. (12); Table 2"}],"minor_comments":[{"comment":"The running header 'Graph-Structured RubricsANTGROUPRESEARCH' appears on multiple pages; please clean this up in the final version.","section":"General formatting"},{"comment":"The '+X.XX' annotations for MAE do not have a consistent sign convention; a plus sign is used for both an increase and a decrease in MAE. Add a legend or use a signed difference in the direction of improvement.","section":"Tables 1 and 3"},{"comment":"Standard deviations are reported only in the text; include them in the tables or an appendix for reproducibility.","section":"Metrics and provenance"},{"comment":"No code or artifact link is provided; since replayability is a stated contribution, a public implementation would help reviewers and users verify the claim.","section":"Implementation protocol"},{"comment":"Table 4 only includes HelpSteer2; the cross-model transfer claim would be strengthened by at least one additional dataset, ideally including a pairwise benchmark.","section":"Cross-Model Sensitivity"}],"recommendation":"major_revision","confidential_remarks":"The semantic-validation gap is the main risk. The manuscript is within scope and the core idea is promising; I would not recommend rejection because the gap can be addressed in revision with a semantic audit and added uncertainty quantification. The citation coverage appears adequate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper introduces an abstraction worth taking seriously: GSR compiles a rubric, before seeing any responses, into a typed DAG with criterion nodes, operators (TRANSFORM, REDUCE, GATE), named ports, and a task-specific Readout. That cleanly separates criterion elicitation from composition, makes the policy explicit and replayable, and is a real step beyond flat-rubric or checklist baselines. The same-trace ablation, where weighted aggregation reuses the same criterion-level judgments, is a good way to isolate the effect of graph composition. The related-work survey is broad and the distinction from DAGMetric and RULERS is plausible.\n\nThe soft spots are significant, though not fatal to the framework. The first is load-bearing: the paper states in the Compile section that it accepts the first structurally valid graph and that \"whether the accepted graph captures the intended meaning of the natural-language rubric is tested through downstream agreement.\" Structural validation checks acyclicity, types, ports, and reachability, but it cannot detect a gate wired to the wrong criterion or a reduction that averages instead of capping. So the headline numbers are evidence about the distribution of graphs this compiler produces, not evidence that the rubric's intended policy is being executed. The stress-test note is right on this point.\n\nSecond, the margins are modest on several datasets (0.62 and 0.99 points exact agreement) and there are no significance tests or confidence intervals, just run-level standard deviations. Third, the pairwise primary metric counts ties and invalid outputs as errors, which systematically favors GSR's no-tie policy; the paper's own Valid Accuracy numbers show TICK and CheckEval doing better on valid decisions. Fourth, no code, data, or prompts are released. Fifth, cross-model results are mixed: on GLM-4.7, GSR's exact agreement is 1.04 points worse than Prometheus-style, so the benefit is backbone-dependent.\n\nDespite all this, the paper deserves serious peer review. The abstraction is useful, the empirical work is honest — the authors explicitly acknowledge the semantic-verification gap rather than burying it — and the experiments are well-designed for isolating composition. I would send it to a strong venue, asking reviewers to focus on the semantic-fidelity issue and the tie-policy metric. A conditional accept would be defensible if the authors add a human or automated semantic check on the compiled graphs and release code. Engage with this, but don't trust the numbers until the semantic check is in.","headline":"A genuinely new abstraction for LLM rubric composition, but its central semantic-fidelity claim is explicitly untested — the reported gains could come from arbitrary graphs rather than faithful rubric compilation.","tokens_in":13689,"tokens_out":3258,"would_cite":true,"duration_ms":28712,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Rubric rules compiled into typed graphs before judging lift LLM score agreement by up to 6.75 points.","keywords":["LLM-as-a-judge","rubric compilation","typed evaluation graph","pointwise scoring","pairwise preference","auditability","criterion composition","evaluation DAG"],"falsifier":"Compile a rubric whose natural-language rules order a safety cap before a reduction, then inspect the accepted graph: if the first validator-approved program wires the gate to the wrong port while downstream agreement on a held-out set stays high, the assumption that structural validity suffices for faithful composition is falsified; conversely, showing that a semantic check changes the accepted graph and shifts agreement would quantify the exposure.","tokens_in":12707,"feed_emoji":"📊","tokens_out":8308,"duration_ms":64049,"temperature":0.7,"pith_summary":"This paper argues that the composition rules in an evaluation rubric—how criterion-level judgments combine into a final score or preference—should be compiled into an explicit typed graph before any candidate response is seen, rather than left inside the prompt for the LLM to execute implicitly. It introduces Graph-Structured Rubrics (GSR), which compiles a rubric into a response-independent directed acyclic graph with criterion nodes, deterministic TRANSFORM, REDUCE, and GATE operator nodes, and a unique sink, then executes that graph in topological order and maps the sink to an output through a task-specific Readout. Across four pointwise datasets and two pairwise benchmarks, the paper reports that GSR under GPT-OSS-120B achieves the highest exact score agreement—0.62 to 6.75 percentage points over Prometheus-style scoring—and the highest end-to-end pairwise accuracy under native tie and abstention policies. The point is not that criterion judgments improve, but that the route they take to a final decision becomes explicit, deterministic, and replayable.","feed_headline":"Rubric graphs lift LLM judge agreement up to 6.75 points","feed_subtitle":"Compiling rubrics into replayable typed graphs beats flat prompt rubrics on six benchmarks.","key_machinery":"The central object is the typed evaluation graph (a DAG) compiled from the rubric before candidate responses are observed: criterion nodes produce judgments, TRANSFORM/REDUCE/GATE operator nodes compose them through named, slot-ordered ports, and a unique sink feeds a task-specific Readout contract $R_\\tau$. A deterministic validator rejects cycles, missing ports, arity mismatches, and type-incompatible routes; execution flows in topological order and records an audit trace that makes composition replayable. The same graph executes pointwise ($m=1$) and pairwise ($m=2$) tasks; only candidate arity, the task contract, and Readout differ.","core_discovery":"The central claim is that making the composition policy of a rubric explicit as a typed, response-independent graph improves the final decision of an LLM judge without changing criterion-level judgments. GSR compiles the rubric into a program $(G, R_\\tau)$ in which criterion nodes emit candidate-aligned judgments, operator nodes (TRANSFORM, REDUCE, GATE) combine them through named ports, and the unique sink produces internal quality scores that a deterministic Readout converts to a native score or a winner/tie preference. Compilation validates acyclicity, port arity, reachability, and type compatibility, accepting the first structurally valid graph without human semantic screening. The paper's experiments show that this composition layer yields higher exact agreement than direct scoring or flat weighted aggregation on the same criterion traces, and that the same graph serves both pointwise and pairwise evaluation with only candidate arity and Readout differing.","pith_inferences":["A testable extension the paper does not run would add semantic validation of the compiled graph—for example re-deriving the natural-language rule from the graph or probing gate placement—to test whether the first structurally valid program is also the intended one.","The audit trace opens a use the paper leaves implicit: converting recorded criterion judgments into preference pairs under a declared tie policy, which could make GSR a substrate for reward-model training data rather than only a final judge.","The Readout contract's $\\epsilon_\\tau$ and $resolve_\\tau$ are an implicit design lever: varying the tie threshold trades abstention against forced choice, and the paper's reported accuracy comparisons depend on this native policy choice.","Because cross-model results show exact-agreement gains reversing (positive with one backbone, negative with another) while Within-1 Accuracy improves in both, a direct probe of the composition layer's contribution would feed gold or human criterion-level judgments through the same graph and measure how much of the agreement gain survives."],"forward_implications":["Rubric composition becomes inspectable and replayable: the audit trace records every criterion judgment, operator parameter, and slot-ordered input, so any reported score can be deterministically replayed from the recorded judgments.","A single compiled graph serves both pointwise scoring and pairwise preference, with only the Readout changing, so the same composition policy carries across task types.","Explicit graph composition shifts final ordinal boundaries even when criterion-level judgments are held fixed; the ablation reusing identical traces shows this, and the paper is explicit that GSR is a controlled composition path rather than a uniform correction of the score distribution.","The benefit is backbone-conditioned: on HelpSteer2 the exact-agreement gain over Prometheus-style is 0.25 points with one judge model and negative 1.04 points with another, so GSR controls composition but does not repair weaker criterion judgments.","In pairwise evaluation the graph interface nearly eliminates non-decisions: GSR kept 99.87--99.91% coverage with a near-zero tie/invalid rate, while checklist baselines tied 19.57--34.66% of cases."],"supporting_citations":[{"why":"supplies the Prometheus-style scoring baseline that GSR is compared against on pointwise datasets","marker":"[Kim et al., 2024b]"},{"why":"supplies the G-Eval structured-reasoning baseline and the sensitivity discussion for LLM judges","marker":"[Liu et al., 2023]"},{"why":"supplies the MT-Bench pairwise benchmark and the documented position/length sensitivities of LLM judges","marker":"[Zheng et al., 2023]"},{"why":"supplies the RubricBench pairwise benchmark with released instance-level checklist rubrics","marker":"[Zhou et al., 2026]"},{"why":"supplies the HelpSteer2 pointwise dataset used for the main and cross-model results","marker":"[Wang et al., 2024]"},{"why":"supplies the SummEval Relevance pointwise dataset","marker":"[Fabbri et al., 2021]"},{"why":"supplies the BiGGen pointwise dataset with instance-level criteria","marker":"[Kim et al., 2025]"},{"why":"supplies the UltraFeedback pointwise dataset combined with TruthfulQA responses","marker":"[Cui et al., 2024]"},{"why":"supplies the OpenRubric baseline in the pairwise comparison","marker":"[Liu et al., 2025]"}],"fun_headline_variants":["Rubric graphs lift LLM judge agreement up to 6.75 points","Graph-structured rubrics boost LLM judge agreement by 6.75 points","Compile rubrics into graphs: LLM judges improve by 6.75 points","Typed graph rubrics: LLM judges gain 6.75-point agreement"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's success depends on the first structurally valid graph the LLM compiler produces actually encoding the rubric's intended composition, because compilation checks only structure and types, never meaning.","fun_headline_variants_meta":{"raw":{"variants":["Rubric graphs lift LLM judge agreement up to 6.75 points","Graph-structured rubrics boost LLM judge agreement by 6.75 points","Compile rubrics into graphs: LLM judges improve by 6.75 points","Typed graph rubrics: LLM judges gain 6.75-point agreement"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001063,"raw_usage":{"total_tokens":4433,"prompt_tokens":898,"completion_tokens":3535,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":3449}},"tokens_in":514,"tokens_out":3535,"duration_ms":22672,"temperature":1.0,"reasoning_tokens":3449,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:16:30.264144+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compile a rubric whose natural-language rules order a safety cap before a reduction, then inspect the accepted graph: if the first validator-approved program wires the gate to the wrong port while downstream agreement on a held-out set stays high, the assumption that structural validity suffices for faithful composition is falsified; conversely, showing that a semantic check changes the accepted graph and shifts agreement would quantify the exposure.","supporting_citations":[],"review_version":1}