{"id":"7293d4f0-ddcf-4b40-ad2d-9217dabf7cd3","arxiv_id":"2603.17145","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.5,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"REAL optimizes LLMs as numeric judges via a policy-dependent regression RL objective that jointly explores reasoning chains and refines continuous score predictions.","lead":"REAL is a reinforcement-learning method that trains LLMs to score other models' outputs with continuous regression rewards instead of binary right/wrong signals. It may produce more reliable automated judges for ranking AI systems, especially when scores must track human ordinal preferences.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Claimed optimality of the generalized policy-gradient for a policy-dependent correlation objective cannot be audited from the abstract; bias in the CoT/score decomposition would undermine attribution of the reported gains.","rationale":"The reader's weakest_assumption correctly isolates the unverifiable correctness of the generalized policy-gradient decomposition as the critical soft spot. With only the abstract available, no stronger internal inconsistency, circularity, or experimental artifact can be demonstrated, nor can the numerical gains be audited. The concern is therefore the same: the load-bearing technical step (unbiased decomposition of a policy-dependent correlation objective) is uncheckable, so the verdict remains UNVERDICTED at low confidence. No red-flag contradiction appears in the abstract itself; the risk is purely that the claimed proof and estimator may not survive scrutiny. The concrete re-derivation check above would settle whether the concern lands once the full text is in hand.","tokens_in":2169,"tokens_out":523,"duration_ms":16575,"concrete_test":"Once the full paper is available, independently re-derive the generalized policy-gradient starting from the correlation objective (Pearson or Spearman) under the paper's autoregressive sampling model, without presupposing the two-term decomposition. Check whether any residual policy-dependent bias term remains when the numeric score is generated conditionally on the CoT. If a non-zero bias term appears, the optimality claim and the causal attribution of the reported gains to REAL are unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on a generalized policy-gradient estimator that is asserted to correctly optimize an explicitly policy-dependent regression objective and to be optimal for Pearson/Spearman. The abstract states that standard policy gradients are invalidated by this dependence, yet that the estimator 'naturally decomposes' optimization into independent CoT-exploration and score-refinement terms. No estimator form, derivation, or bias analysis is supplied. If the decomposition assumes a separation between trajectory distribution and final-score gradient that fails under autoregressive CoT sampling (or leaves a residual policy-dependent term), the estimator is biased in the LLM-as-a-Judge regime; the optimality guarantee then does not hold and the reported gains (+8.40 Pearson / +7.20 Spearman over SFT, etc.) cannot be attributed to the claimed mechanism. This is the single load-bearing step: experimental outperformance and OOD generalization claims inherit their force from the estimator being correct.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The manuscript proposes REAL (Regression-Aware Reinforcement Learning), an RL framework for training LLM-as-a-Judge models that assign continuous numeric scores. It argues that standard RL with binary rewards ignores ordinal structure in regression-style judging, while existing regression-aware methods are limited to SFT and thus under-explore reasoning paths. REAL optimizes continuous regression rewards (Pearson/Spearman) via a generalized policy-gradient estimator intended to handle an explicitly policy-dependent objective; the estimator is claimed to decompose into CoT-trajectory exploration and final-score refinement, and the resulting policy is asserted to be optimal for correlation metrics. Across 8B–32B models the abstract reports consistent gains over regression-aware SFT and standard binary-reward RL, with better OOD generalization; on Qwen3-32B the stated improvements are +8.40 Pearson / +7.20 Spearman over SFT and +18.30 / +11.20 over the base model.","tokens_in":2428,"tokens_out":1173,"duration_ms":21686,"significance":"If the generalized estimator is unbiased for the policy-dependent correlation objective and the optimality claim for Pearson/Spearman holds under stated assumptions, REAL would fill a clear methodological gap between regression-aware SFT and binary-reward RL for LLM judges, with direct practical value for automated evaluation. The reported effect sizes are large and the OOD generalization claim, if substantiated, would further strengthen impact. The abstract’s emphasis on a principled decomposition and an optimality result is a genuine strength if those results are carefully derived (or machine-checked) in the full paper; significance remains conditional on those technical claims surviving scrutiny.","major_comments":[{"comment":"The load-bearing technical claim is that a generalized policy-gradient estimator correctly optimizes an explicitly policy-dependent regression objective (invalidating standard policy gradients) and is optimal for Pearson/Spearman, while ‘naturally decomposing’ into independent CoT-exploration and score-refinement terms. From the abstract alone there is no estimator form, derivation, or bias analysis. If the decomposition assumes a separation between trajectory distribution and final-score gradient that fails under autoregressive CoT sampling, or leaves a residual policy-dependent term, the estimator is biased in the LLM-as-a-Judge regime; the optimality guarantee and attribution of the reported gains would then not hold. This step must be fully derived and bias-checked in the manuscript before the central claim can be accepted.","section":"Abstract (central claim / estimator)"},{"comment":"The abstract asserts the method is ‘proven to be optimal for correlation metrics’ without a proof sketch, assumptions, or scope (e.g., policy class, reward form, finite-sample vs. population). Optimality for Pearson/Spearman is not automatic for arbitrary regression-style rewards; the manuscript must state the precise theorem, the conditions under which it holds, and whether the implemented estimator matches the object of the proof. Without that, the optimality language overclaims relative to what can be audited from the abstract.","section":"Abstract (optimality claim)"},{"comment":"Reported gains (+8.40 Pearson / +7.20 Spearman over SFT; +18.30 / +11.20 over base on Qwen3-32B) are large but, in the abstract, lack error bars, seed counts, statistical tests, and ablations that isolate the two claimed components (CoT exploration vs. score refinement) from ordinary RL hyper-parameters (learning rate, KL coefficient, reward scaling). Without those controls, outperformance cannot be attributed to the generalized estimator rather than training recipe differences. The OOD generalization claim similarly needs a defined protocol and baselines before it can support the central mechanism.","section":"Abstract (experiments / Qwen3-32B results)"}],"minor_comments":[{"comment":"The abstract packs several strong claims (policy-dependence invalidating standard PG; natural decomposition; optimality; large multi-scale gains; OOD superiority) into a short space. Once the full text is available, a short ‘claims checklist’ early in the paper mapping each claim to a theorem/equation and a table would help readers audit load-bearing steps.","section":"Abstract"},{"comment":"Clarify early what ‘regression rewards’ concretely are (Pearson, Spearman, MSE, or a mixture) and whether the same estimator is used for all reported metrics, to avoid conflating training objective with evaluation metric.","section":"Abstract"},{"comment":"When the full manuscript is reviewed, ensure the related-work section positions the ‘generalized policy gradient’ relative to existing policy-gradient variants for non-standard or trajectory-dependent rewards so novelty is explicit.","section":null}],"recommendation":"uncertain","confidential_remarks":"This is an abstract-only review; the full text was not available. Confidence is therefore low. The single load-bearing risk is the correctness/bias of the generalized policy-gradient estimator for a policy-dependent correlation objective under autoregressive CoT sampling—if that derivation is sound and ablations isolate the two components, the paper is likely a solid major_revision or better candidate; if not, the large reported gains cannot be attributed to the claimed mechanism. I recommend obtaining the full manuscript (and, if possible, code) before a final decision. Scope fit for cs.LG / LLM evaluation venues appears reasonable if the technical claims hold."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing to know: this paper attacks a real gap. Binary-reward RL for LLM-as-a-Judge throws away ordinal structure, and regression-aware SFT does not explore CoT. REAL tries to do both—optimize continuous regression rewards (Pearson/Spearman) with RL—via a generalized policy-gradient estimator that is claimed to handle the policy-dependent objective and to be optimal for correlation metrics. Reported lifts on Qwen3-32B (+8.4 Pearson / +7.2 Spearman over SFT, larger over base) are big enough that, if real, practitioners building judges and leaderboards will care.\n\nWhat looks new and solid from the abstract: framing the regression objective as explicitly policy-dependent (so standard PG fails), then decomposing into CoT exploration plus final-score refinement; multi-scale experiments (8B–32B) and an OOD generalization claim. That is a natural next step after regression-aware SFT and binary RL, but the joint treatment and the optimality claim for correlation are the actual contribution, not just another reward tweak.\n\nSoft spots, in proportion. We only have the abstract. The estimator form, derivation, and bias analysis are not here. The stress-test concern is fair: if the CoT/score decomposition leaves a residual policy-dependent term under autoregressive sampling, the optimality story and the attribution of the gains both weaken. That is the single load-bearing step. Secondary: free RL hyperparameters, no error bars or ablations visible yet. None of that is a red flag of circularity or self-dealing; it is simply unaudited math and incomplete reporting at this stage.\n\nWho it is for: people who train or evaluate LLM judges, synthetic-data pipelines, and alignment reward models. Not a theory paper for pure RL theorists unless the full derivation is clean. It deserves a serious referee—send it to review. If the estimator checks out and the gains hold with proper ablations and error bars, cite it; if the decomposition is hand-wavy, revise hard. I would bring it to reading group only after the full PDF is in hand, not on abstract alone.","headline":"Useful practical recipe for regression-aware RL judges; the generalized policy-gradient claim is the load-bearing piece and cannot be audited from the abstract alone.","tokens_in":3008,"tokens_out":537,"would_cite":false,"duration_ms":10790,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"REAL optimizes continuous regression rewards for LLM-as-a-Judge via a generalized policy-gradient estimator, proving optimality for correlation metrics and beating both SFT and binary RL.","keywords":["LLM-as-a-Judge","regression-aware reinforcement learning","policy gradient","correlation metrics","chain-of-thought","ordinal scoring","generalized policy gradient"],"falsifier":"Train REAL, regression-aware SFT, and binary-reward RL on the same judge data, then measure Pearson and Spearman on a held-out out-of-domain benchmark; if REAL does not clearly exceed both baselines, the central claim fails.","tokens_in":3048,"feed_emoji":"⚖️","tokens_out":775,"duration_ms":14928,"temperature":0.7,"pith_summary":"This paper claims that LLM-as-a-Judge training should reward continuous numeric scores rather than binary correctness, because ordinal structure matters: predicting 4 when the truth is 5 is far better than predicting 1. Standard RL collapses that structure into 0-1 rewards, while existing regression-aware methods stop at supervised fine-tuning and cannot explore better reasoning. The authors introduce REAL, a reinforcement-learning framework that optimizes regression rewards with a generalized policy-gradient estimator. The estimator is argued to be optimal for correlation metrics and to decompose naturally into exploring chain-of-thought trajectories and refining the final score. Across 8B–32B models it consistently improves Pearson and Spearman correlation over both regression-aware SFT and binary-reward RL, with larger gains on out-of-domain benchmarks.","feed_headline":"Regression-aware RL lifts LLM judges +8 Pearson over SFT","feed_subtitle":"Continuous rewards and CoT exploration raise correlation and out-of-domain generalization on 8B–32B models.","key_machinery":"The generalized policy-gradient estimator for an explicitly policy-dependent regression objective. It decomposes the gradient into two complementary terms: exploration over chain-of-thought trajectories and regression-aware refinement of the final numeric score.","core_discovery":"REAL is a regression-aware RL method for LLM judges that optimizes continuous rewards; its generalized policy-gradient estimator is proven optimal for correlation metrics and decomposes optimization into CoT exploration plus score refinement, delivering consistent gains over regression-aware SFT and binary RL.","pith_inferences":["The same CoT-plus-score decomposition may apply to other graded LLM outputs such as rating, ranking, or calibration.","Binary-reward RL may systematically under-train models on any task whose natural evaluation is ordinal rather than pass/fail.","If the estimator is unbiased, other policy-dependent continuous objectives (ranking losses, expected calibration error) could be folded into LLM RL the same way."],"forward_implications":["LLM judges trained with continuous regression rewards reach higher Pearson and Spearman correlation with human scores than those trained with binary RL or regression SFT.","The same training yields stronger generalization on out-of-domain evaluation benchmarks.","Regression objectives can be optimized inside an RL loop without collapsing to 0-1 accuracy signals.","At 32B scale the method produces double-digit correlation gains over the base model."],"fun_headline_variants":["REAL: regression-aware RL for LLM judges beats SFT by +8 Pearson","Continuous rewards let RL refine CoT and scores for better judging","Policy gradient for ordinal rewards lifts LLM-as-Judge correlations","REAL dual optimizes CoT paths and score precision over binary RL","Regression-aware RL generalizes better than SFT on LLM evaluators"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The generalized policy-gradient estimator correctly decomposes a policy-dependent regression objective into independent exploration and score-refinement terms without introducing bias that would undermine the claimed optimality for correlation metrics.","fun_headline_variants_meta":{"raw":{"variants":["REAL: regression-aware RL for LLM judges beats SFT by +8 Pearson","Continuous rewards let RL refine CoT and scores for better judging","Policy gradient for ordinal rewards lifts LLM-as-Judge correlations","REAL dual optimizes CoT paths and score precision over binary RL","Regression-aware RL generalizes better than SFT on LLM evaluators"]},"model":"grok-4.5","effort":"low","cost_usd":0.00496,"raw_usage":{"total_tokens":1468,"prompt_tokens":866,"num_sources_used":0,"completion_tokens":95,"cost_in_usd_ticks":49600000,"prompt_tokens_details":{"text_tokens":866,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":507,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":866,"tokens_out":95,"duration_ms":5386,"temperature":1.0,"reasoning_tokens":507,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T23:18:59.531110+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train REAL, regression-aware SFT, and binary-reward RL on the same judge data, then measure Pearson and Spearman on a held-out out-of-domain benchmark; if REAL does not clearly exceed both baselines, the central claim fails.","supporting_citations":[],"review_version":1}