{"id":"be7132eb-56a5-4215-bf36-3d2ed2be5b57","arxiv_id":"2510.15514","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Removing minimum feedback arc sets from LLM-judge preference graphs yields small benchmark gains in RLAIF, but claims outrun the evidence.","lead":"This paper proposes cleaning up AI-judge preference feedback by deleting cyclic inconsistencies before using it to train language models. It reports small benchmark gains over pairwise and ranking baselines, but the evidence lacks error bars and the abstract and main text describe different frameworks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DGR's deconfliction step is under-specified: minimum FAS is not unique, so Eq. (3) can return different reward vectors for the same raw judgments.","rationale":"The reader's weakest assumption — that preference cycles are stochastic noise rather than genuine intransitivity — is reasonable but not the most load-bearing for the central claim. Even if cycles are genuine, DGR can be viewed as a Kemeny-style rank aggregation: the minimum-FAS solution is a defensible consensus ranking, so the noise assumption is not strictly necessary for the method to produce a useful reward. The more serious problem is that DGR as specified is incomplete: in a 3-cycle, three different minimum FAS choices are equally 'optimal' under the paper's cardinality criterion, and each yields a different reward vector via Eq. (3). The paper provides no tie-breaking rule, no description of how the exhaustive search selects among optima, and no averaging over optimal solutions. Since the reward is exactly the net-win score of the chosen DAG, this ambiguity can reorder the reward signal and change the RL objective. The availability of code makes this testable: if different tie-breaks change the final benchmark scores, then the reported improvements are not attributable to the stated graph-theoretic optimization criterion. This is more fundamental than the absence of variance bars or the abstract/body mismatch; it concerns whether the central construction is well-defined. The appropriate verdict remains CONDITIONAL, because the issue is addressable by specifying or averaging the tie-breaking, and the released code may already resolve it in a particular way — but the manuscript as written does not.","tokens_in":17689,"tokens_out":11013,"duration_ms":98641,"concrete_test":"Run the released DGR code on a fixed G=3 preference graph with judgments A≻B, B≻C, C≻A, and enumerate all three minimum-FAS solutions. If the deconflicted net-win score vectors from Eq. (3) differ across the three choices (or the code cannot be configured to produce all three by a documented parameter), Stage 2 is under-specified. As a second check, rerun the Arena-Hard GRPO setup with two different min-FAS tie-breaks (e.g., reversed edge iteration or a different solver ordering) and compare the reported peak score of 52.7; if the peak moves by more than about 1 point, the claimed 'optimal resolution' advantage is not robust to the unspecified tie-breaking.","verdict_should_be":"UNCHANGED","load_bearing_attack":"§3.2.1 Stage 2 removes 'a minimum feedback arc set' to transform the preference graph into a DAG. For any graph containing cycles, the minimum FAS need not be unique; in a semicomplete digraph with a 3-cycle A≻B≻C≻A, every one of the three edges is a minimum FAS. The three resulting DAGs are B≻C≻A, C≻A≻B, and A≻B≻C, so the Stage-3 net-win scores in Eq. (3) are respectively (B=1,C=0,A=-1), (C=1,A=0,B=-1), and (A=1,B=0,C=-1) — genuinely different reward vectors. The paper never specifies a tie-breaking rule or an aggregation over all optimal FAS sets. Consequently DGR's reward signal is not a well-defined function of the raw judgments; the 'DGR (Optimal Resolution)' entry in Table 2 may owe its gain to an arbitrary deterministic choice in the implementation rather than to the stated optimality criterion. This directly threatens the reproducibility of the central stability/performance claim and the interpretation of the RandomResolve/ReverseResolve ablation.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a diagnostic metric, Conflict Detection Rate (CDR), and a reward-generation front-end, Deconflicted Graph Rewards (DGR), for RLAIF. DGR constructs a directed preference graph from all pairwise LLM judgments over G responses, removes a minimum feedback arc set to make the graph a DAG, and assigns each response a net-win score (out-degree minus in-degree). These scores are normalized into advantages and used inside GRPO or GSPO. Experiments on Qwen3-14B/8B trained on 1,000 WildChat queries and evaluated on Arena-Hard, MT-Bench, and WritingBench report gains over pointwise, listwise, pairwise, and ELO baselines, along with ablations over conflict-resolution strategy, judge prompt, judge model, and graph size.","tokens_in":18009,"tokens_out":6102,"duration_ms":54679,"significance":"If the central claim is confirmed, DGR would be a cheap, model-agnostic preprocessing layer for preference-based RL, and CDR would be a useful diagnostic for judge consistency. The paper has real strengths: the graph-construction idea is simple and modular; the final evaluation uses GPT-4.1 and Claude-3.7 as judges, different from the Qwen3-32B training judge, which avoids the most direct circularity; and the code is promised to be public. However, the evidence as reported is not yet sufficient. The reward signal is not a well-defined function of the raw judgments because the minimum FAS is generally non-unique and no tie-breaking rule is given; the headline results are 'peak scores' from two runs with no variance or significance testing; and the paper's central assumption that cycles are stochastic noise rather than genuine intransitivity is never validated. These issues are substantive but appear fixable within the manuscript's scope.","major_comments":[{"comment":"DGR's reward signal is not well-defined as stated. A minimum feedback arc set is generally non-unique. For a 3-cycle A≻B≻C≻A, deleting any one edge is a minimum FAS, and the three resulting DAGs give net-win vectors (A=1,B=0,C=-1), (A=-1,B=1,C=0), and (A=0,B=-1,C=1), respectively. The paper does not specify a tie-breaking rule, an aggregation over all optimal FAS sets, or a deterministic implementation. Therefore Eq. (3) can return different reward vectors for identical raw judgments, and the 'DGR (Optimal Resolution)' entry in Table 2 is not reproducible from the text. Please specify a canonical tie-break, average over all optimal FAS sets, or provide the exact deterministic algorithm and seed used in the experiments.","section":"§3.2.1, Stage 2, and Eq. (3)"},{"comment":"The paper reports 'the peak score achieved across all training steps from two independent experimental runs.' Peak-over-training is an optimistically biased statistic rather than an estimate of expected performance, and with n=2 there is no variance, confidence interval, or significance test. Table differences of 0.5-1.5 points are therefore not sufficient to support claims such as 'significantly improves' or 'dominant performance.' Please report full training curves and mean ± standard deviation or standard error over at least 3-5 seeds, with paired significance tests or bootstrap intervals for the headline comparisons.","section":"§4.1, Evaluation Protocol; Tables 1-3"},{"comment":"The correlation analysis uses only four prompts (P2-P5). With n=4, the reported correlations, including r=0.98 between CDR and accuracy and the near-zero r for DGR, are highly sensitive to a single data point and cannot support the paper's 'accuracy-consistency dilemma' or 'DGR is uncorrelated with signal quality' conclusions. Moreover, P5 is also the prompt used in Table 1 and Table 2, so the four-point set is not an independent spread. Please add more prompt conditions or replace the correlation claim with paired per-prompt comparisons and confidence intervals.","section":"§4.4.2, Table 3"},{"comment":"The method's justification is that observed preference cycles arise mainly from stochastic measurement error, not genuine intransitivity. This is stated as an assumption but never tested. If LLM or human preferences are legitimately non-transitive (e.g., multi-criteria trade-offs), then removing a minimum FAS deletes true signal and the reported gains could be an artifact of discarding disagreeing evidence. The paper should validate this assumption directly, for example by comparing the judge accuracy of removed FAS edges versus retained edges on RewardBench2 or another ground-truth preference dataset, and should discuss what happens when the noise model is violated.","section":"Abstract and §3.2.1"}],"minor_comments":[{"comment":"The abstract refers to 'Topological Consensus Rewards (TCR)' and 'Cycle Incidence Rate (CIR),' while the full text consistently uses 'Deconflicted Graph Rewards (DGR)' and 'Conflict Detection Rate (CDR).' This suggests a version mismatch and should be fixed.","section":"Abstract vs. main text"},{"comment":"CDR depends on the number of candidates G in each sample, but the paper does not state G for the RewardBench2 evaluation. The reader must infer that one chosen plus three rejected responses gives G=4. Please state this explicitly.","section":"§4.1 and Figure 1"},{"comment":"The ELO baseline iterates over 'each comparison pair' without specifying the update order. Different orderings can lead to different final ratings. Please state the exact ordering or use a batch update.","section":"Appendix C.1.3, Algorithm 7"},{"comment":"The pointwise baseline maps scores directly to rewards, but the DGR/ELO/PREF methods normalize within a group. The comparison would be cleaner if all methods used the same normalization before computing advantages.","section":"§4.2 and Table 1"},{"comment":"DGR-RandomResolve and DGR-ReverseResolve use random edge selection without a specified seed. Since the reader cannot reproduce these ablations, please provide seeds or a deterministic protocol.","section":"§4.4.1, Table 2"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and potentially publishable, but the paper needs to resolve the minimum-FAS non-uniqueness issue and substantially strengthen the empirical reporting. I also noticed the abstract/body terminology mismatch, which may indicate a hasty revision. If the implementation can be made fully deterministic and the results can be re-run with variance reporting, the paper would be much easier to evaluate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe paper is not what its front matter says it is. The abstract describes 'Topological Consensus Rewards' and 'Cycle Incidence Rate'; the body actually proposes 'Deconflicted Graph Rewards' and 'Conflict Detection Rate', and the arXiv title is different again. Read the body, not the abstract.\n\nWhat's genuinely new: they apply minimum feedback arc set / Slater-style deconfliction inside the RLAIF loop as a reward front-end for GRPO and GSPO. That is a sensible, cheap, model-agnostic idea, and the CDR diagnostic is useful. They also evaluate with different judge models from the training judge, which is good practice.\n\nBut the method has a load-bearing gap. Minimum FAS is not unique, and for a 3-cycle every single edge is a minimum FAS, so Eq. (3) can return three different reward vectors for the same raw judgments. The paper never specifies a tie-break or an aggregation over optimal FAS sets. That means 'DGR (Optimal Resolution)' in Table 2 is not a well-defined function of the raw judgments, and the difference between RandomResolve and Optimal may be an artifact of whatever deterministic choice the code makes. This is not a minor issue; it is the central mechanism.\n\nThe empirical case is also thinner than the prose. Results are peak scores from two runs with no variance or significance, training uses only 1,000 WildChat queries, and some cells in Table 1 contradict the 'consistently outperforms' claim (DGR-GSPO WritingBench is below PREF, DGR-GRPO Math below ELO). The core assumption that cycles are noise rather than genuine intransitivity is never tested—they do not check whether removed edges were actually wrong.\n\nWhat holds up: the integration with GRPO/GSPO is clearly described, the baselines are reasonable, and the graph-theoretic idea is well grounded in prior work, which they cite. The paper would be a useful contribution to the RLAIF/alignment subfield once these issues are addressed.\n\nWho is this for: anyone working on RLAIF reward design or LLM judge evaluation. It deserves a serious referee—the idea is real, the flaws are fixable, and a good review could force the authors to make the method deterministic and report proper statistics. I would accept for review, not desk reject, and ask for major revision.\n\nBest","headline":"A real idea—FAS deconfliction as a reward front-end—but as submitted the method is under-specified (minimum FAS is non-unique) and the empirical claims outrun the evidence.","tokens_in":18515,"tokens_out":3019,"would_cite":false,"duration_ms":26738,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that preference cycles in LLM judge feedback are measurement noise, and removing a minimum feedback arc set before policy optimization yields more stable, better-performing RLAIF than pairwise, listwise, pointwise, or ELO","keywords":["RLAIF","LLM judge","preference cycles","feedback arc set","directed acyclic graph","reward signal","transitivity","training stability"],"falsifier":"Collect many repeated, re-prompted judgments for the same response triples and check whether the cycle edges that DGR removes survive at high confidence. If a meaningful fraction are stable and reproducible, or if a dataset with known genuine intransitivity shows DGR hurting rather than helping downstream performance, the noise model and the central claim would be falsified.","tokens_in":17591,"feed_emoji":"🔄","tokens_out":8575,"duration_ms":69337,"temperature":0.7,"pith_summary":"This paper is trying to establish that preference cycles—A beats B, B beats C, C beats A—in LLM judge feedback are largely stochastic measurement errors, and that deleting the smallest set of edges needed to break them turns conflicted judgments into a reward signal that makes reinforcement learning from AI feedback (RLAIF) more stable and more accurate. It introduces a diagnostic metric, CDR, that measures how often a judge's pairwise verdicts contain a cycle, and a reward front-end, DGR, that builds a directed preference graph, removes a minimum feedback arc set to make it acyclic, and scores each response by net wins (out-degree minus in-degree) in the cleaned graph. On Arena-Hard, MT-Bench, and WritingBench, DGR integrated with GRPO and GSPO outperforms pointwise, listwise, pairwise win-rate, and ELO baselines, and remains stable across different judge models and group sizes. A sympathetic reader would care because the fix is cheap and modular: it does not require retraining the judge or changing the policy optimizer.","feed_headline":"Preference cycles are noise; deleting them improves RLAIF","feed_subtitle":"A graph front-end turns conflicted judge votes into a DAG and beats pairwise, listwise, pointwise, and ELO rewards.","key_machinery":"Deconflicted Graph Rewards (DGR): a three-stage reward front-end that (1) constructs a semicomplete directed preference graph from all pairwise judge judgments, with ties contributing no edge; (2) converts the graph into a Directed Acyclic Graph by removing a minimum feedback arc set, solved exactly by exhaustive search when the group has at most 10 responses and approximated for larger groups; and (3) assigns each response a reward equal to out-degree minus in-degree in the DAG. These net-win scores are normalized into advantages for group-based optimizers such as GRPO and GSPO. The companion diagnostic, Conflict Detection Rate (CDR), uses strongly connected components of the preference gra","core_discovery":"The paper's central claim is that logical coherence of AI feedback is a separable, addressable dimension of reward quality, and it operationalizes this through topological consistency maximization. Specifically, it reports that current LLM judges produce preference cycles in roughly 1.5 to 6.7 percent of evaluated samples, that these cycles violate the transitivity assumptions underlying preference learning, and that explicitly resolving them by minimum-feedback-arc-set deletion—not by random cycle-breaking or edge reversal—improves downstream policy performance. The resulting reward is a net-win score computed on a DAG, which the paper argues preserves the maximum amount of original prefere","pith_inferences":["A direct test of the paper's noise assumption: re-query the judge on the edges DGR removes, repeatedly and with varied prompts. If removed edges are reproduced at high confidence, some cycles encode genuine preference structure and DGR is deleting real signal rather than noise.","The net-win score weights every surviving edge equally; a natural extension would weight edges by judge confidence or margin, potentially improving reward calibration and training stability further.","CDR could be used adaptively during training—for example, shrinking the comparison group or switching prompts when the measured conflict rate in a batch spikes."],"forward_implications":["DGR can be inserted in front of any group-based preference optimizer, so existing RLAIF pipelines can gain stability without retraining the judge or altering the learning algorithm.","CDR gives a label-free diagnostic that complements accuracy, revealing a trade-off in which more accurate judge prompts tend to produce more preference cycles.","If the noise model is right, reducing CDR during prompt or judge selection should reduce reward noise and improve RL stability in practice.","The ablation shows the resolution strategy matters: random cycle-breaking barely helps and edge reversal can hurt, so the minimum-feedback-arc-set choice carries the improvement.","DGR's advantage over win-rate rewards holds across different judge models and grows with a weaker judge, suggesting the clean-up matters most exactly when judgments are noisiest."],"fun_headline_variants":["Preference cycles are noise; delete them for better RLAIF","Graph trick: resolve judge cycles to improve RLAIF","Topological consensus improves RLAIF via cycle removal","Break judge preference cycles to stabilize RLAIF","Deleting preference cycles improves RLAIF rewards"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The claim rests on treating observed preference cycles as stochastic measurement errors rather than genuine intransitivity; if judges' cycles are real (for instance, reflecting multi-criteria trade-offs), DGR removes true signal and the reported gains would be an artifact of discarding information the model disagreed with.","fun_headline_variants_meta":{"raw":{"variants":["Preference cycles are noise; delete them for better RLAIF","Graph trick: resolve judge cycles to improve RLAIF","Topological consensus improves RLAIF via cycle removal","Break judge preference cycles to stabilize RLAIF","Deleting preference cycles improves RLAIF rewards"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000299,"raw_usage":{"total_tokens":1569,"prompt_tokens":749,"completion_tokens":820,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":741}},"tokens_in":493,"tokens_out":820,"duration_ms":7418,"temperature":1.0,"reasoning_tokens":741,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T09:22:03.039734+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect many repeated, re-prompted judgments for the same response triples and check whether the cycle edges that DGR removes survive at high confidence. If a meaningful fraction are stable and reproducible, or if a dataset with known genuine intransitivity shows DGR hurting rather than helping downstream performance, the noise model and the central claim would be falsified.","supporting_citations":[],"review_version":1}