Pith. sign in

REVIEW 4 major objections 5 minor 13 references

Fusing hidden states of multiple reward models, rather than averaging their scores, produces a better judge of open-ended LLM responses.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 09:06 UTC pith:D3MBOADD

load-bearing objection A cleanly-specified fusion method whose OOD claim is likely inflated by post-hoc readout selection. the 4 major comments →

arxiv 2607.20862 v1 pith:D3MBOADD submitted 2026-07-23 cs.CL

CSPF: A Constrained Shared-Private Fusion Method for Non-Verifiable Preference Evaluation

Hehao Zhang , Danli Wang , Xinyuan Wang , Xuange Gao This is my paper
classification cs.CL
keywords reward model fusionhidden-state representationsnon-verifiable preference evaluationpairwise preference learningshared-private representationLLM evaluatorsout-of-distribution generalization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Open-ended writing, chat, and subjective instruction following have no single correct answer, so evaluating them means combining several loosely aligned human criteria. CSPF argues that the standard interfaces for evaluators—a single holistic reward score, a rubric-conditioned natural-language judgment, or an average of several expert scores—each throw away information that is still present in a reward model's internal representation, just before it is compressed into a score. The paper's proposal is to keep those hidden states, split each into a shared part that aligns across experts and a private part that preserves the expert's own viewpoint, and train a small fusion head on pairwise human preferences. Done this way, CSPF reports the best accuracy among all tested single-expert, multi-expert scalar, and rubric-judge baselines on both a Chatbot Arena-derived validation set and the out-of-distribution PPE benchmark. If right, the result says that the path to better judgment is representation-level integration of many frozen specialists rather than training one larger holistic judge.

Core claim

The central claim is that hidden-state fusion of frozen reward models is a more expressive basis for non-verifiable preference evaluation than the evaluated alternatives: single-expert reward models, scalar-score multi-expert fusion, and rubric-based judges. CSPF projects each expert's hidden readout into a common space, decomposes it into a shared encoder (cross-expert common information) and an expert-private encoder (complementary viewpoint), arranges these as tokens, and passes them through a fusion module; normalized scalar scores are concatenated only as calibration. Training couples a pairwise preference objective with a redundancy-reduction loss that aligns shared representations acr

What carries the argument

Constrained shared-private hidden-state factorization. Each frozen expert contributes a projected hidden-state readout; a shared encoder extracts the cross-expert common component and a private encoder preserves the expert-specific component. The sequence of shared and private tokens is fed to a learned fusion module, with normalized scalar scores concatenated only as auxiliary calibration. Training runs three losses together: a pairwise preference loss for task supervision, a redundancy-reduction objective that aligns shared representations across experts, and a supervised-contrastive objective that separates private representations by source expert.

Load-bearing premise

The load-bearing premise is that the active expert pool (Skywork, OffsetBias, ArmoRM) and the hidden-state readout (0.75L, last-nonpad plus response mean) were fixed before the PPE out-of-distribution results were inspected; if those choices were influenced by the OOD numbers, the reported out-of-distribution advantage is not a fully out-of-sample result.

What would settle it

Retrain CSPF with the expert pool and hidden-state readout chosen using only LM-Arena validation, keeping PPE untouched until the final evaluation; if the accuracy gap over scalar-score baselines disappears, the out-of-distribution claim fails. A complementary test would find a preference benchmark where a scalar-score ensemble over a different, larger expert pool matches or exceeds CSPF's pairwise accuracy while CSPF's chosen configuration does not transfer.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Reward-model ensembling can move from score averaging to learning a fusion head over internal states without retraining the underlying models.
  • Multi-expert evaluation becomes a coupled design problem: the right hidden layer, the active expert pool, and the fusion structure interact, and the paper's pool-ladder experiment shows that adding more experts is not a reliable scaling rule.
  • Per-instance attribution diagnostics show CSPF leans on its strongest general expert but lets specialized experts dominate on particular examples, so hidden-state fusion can support per-instance reliance on complementary criteria.
  • The gains are consistent with intermediate hidden layers carrying preference-relevant information that is compressed away by final scalar outputs.
  • The gap over scalar-score ensembles suggests that multi-expert methods that only combine final scores are limited by the information loss at score compression.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If this generalizes, a practical next step is to use the CSPF reward in downstream pipelines—best-of-N selection, DPO, or RLHF—on an open-ended task; the paper states that this transfer is untested, so a positive result there would consolidate the claim.
  • The shared/private machinery could also be pointed at generative judges, reading their internal states rather than parsing their rubric text, which would broaden the expert pool beyond scalar reward models.
  • A fairer out-of-distribution check would be to fix the expert pool and hidden readout using only in-domain validation before ever looking at PPE; the paper's reported configuration may have been shaped by results on the OOD set, which would weaken the out-of-sample interpretation.
  • The broader pattern suggests that expensive pairwise preference data can supervise a small fusion head over a growing library of frozen specialists, so evaluator improvement would come to depend more on data curation and fusion architecture than on training larger judge models.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes CSPF, a fusion method for non-verifiable preference evaluation that integrates multiple frozen reward models at the level of hidden-state representations rather than scalar scores. Each expert's projected hidden state is decomposed into a shared representation and an expert-private representation; these are fed as a token sequence into a fusion module, and the final reward combines the fused representation with normalized scalar scores as calibration signals. Training uses a Bradley–Terry pairwise objective with a Barlow-Twins-style shared-space loss and a supervised-contrastive private-space loss. The method is evaluated on an LM-Arena validation split and on the PPE out-of-distribution benchmark, reporting the highest LM-Arena validation accuracy (68.04) and PPE off6 accuracy (61.67) among the compared single-expert, rubric, and scalar-score multi-expert baselines. Additional analyses study the hidden-state readout, expert-pool composition, component ablations, and sample-level expert-contribution patterns.

Significance. If the reported results are valid, CSPF would provide useful evidence that hidden-state-level fusion of frozen reward models is a more expressive interface than scalar-score aggregation for non-verifiable preference evaluation. The paper clearly specifies the architecture and training objective, and the component ablation in Table 2 is internally consistent with the narrative. The shared/private factorization and the readout and pool analyses are sensible and potentially informative. The manuscript also includes seed-level standard deviations for the main ablation and a transparent post-hoc Group-Shapley diagnostic. However, the central empirical claim is currently undermined by an unspecified model-selection protocol: both the hidden-state readout and the expert pool appear to have been selected after inspecting PPE results, so the reported PPE advantage may not be genuinely out-of-sample. In addition, the main comparison in Table 1 lacks error bars and significance tests, and the reported numbers are not fully consistent across tables. These issues are load-bearing for the paper's headline claim.

major comments (4)
  1. [§5.3–5.4, Tables 3 and Figure 4] The paper does not state a model-selection protocol that keeps PPE out-of-sample. Section 5.3 reports PPE off6 for many hidden-state readouts, and Section 5.4 reports PPE off6 for many expert pools; the main configuration (0.75L last-nonpad + response mean, S+O+A) is described as a compromise rather than as the choice maximizing LM-Arena validation accuracy. Table 3 shows that the LM-Arena-optimal readout (0.625L last-nonpad) gives PPE off6 = 59.83, only 0.29 points above the LSC baseline (59.54), whereas the reported main configuration gives 61.41 in Table 3 and 61.67 in Table 2. If PPE was used to select the readout or pool, the claimed OOD advantage of 2.13 points over LSC is not out-of-sample. This is load-bearing for the central contribution. Please specify a pre-registered selection rule based only on LM-Arena validation, or report the performance of the configuration chosen withou
  2. [Tables 2 and 3] There is an unexplained inconsistency in the numbers for the main configuration. Table 2 reports CSPF (full model) as 68.04 ± 0.13 LM-Arena, 61.67 ± 0.19 PPE off6, and 69.17 ± 0.13 O/NT. Table 3, described as varying only the hidden-state readout, reports the row corresponding to the main configuration (0.75L last-nonpad + response mean) as 67.83, 61.41, and 69.10. Table 1 reports 68.04 and 61.67. If Table 3 is single-seed, the paper should say so; if it is seed-averaged, the numbers should match Table 2. As written, the reader cannot determine which values are the reliable estimates of the main result, and this directly affects the validity of the headline comparison.
  3. [Table 1] The main comparison table reports single accuracies without error bars, confidence intervals, or significance tests. The difference between CSPF and LSC on PPE off6 is 2.13 points, and the difference between CSPF and the Skywork adapter is 0.50 points on PPE off6. Given that Table 2 shows seed-level standard deviations around 0.13–0.66 on these metrics, the reader cannot tell whether the differences are statistically reliable. In addition, the Skywork LoRA baseline is reported for a single seed (seed 43) while CSPF is averaged over seeds 40, 43, 45; this is not a matched comparison. Please provide paired significance tests with multiple seeds for all baseline rows, or at least report seed-level dispersion for the key comparisons.
  4. [§5.4, Figure 4] The expert-pool ladder analysis is presented as evidence that S+O+A is the best pool, but the same selection-on-PPE concern applies. Figure 4 shows PPE off6 for pools S, S+O, S+O+A, and larger pools, and the choice of S+O+A is justified partly by PPE performance. The paper's stated conclusion that 'expert-pool expansion is a system-level design decision' is reasonable, but if the pool was selected after inspecting PPE, then the main CSPF result in Table 1 is not an honest out-of-distribution evaluation. At minimum, the pool should be chosen using only LM-Arena validation, and the resulting PPE performance should be reported separately from any post-hoc pool analysis.
minor comments (5)
  1. [Table 1, LoRA row] The numerical formatting is broken in the Skywork-V2 LoRA row: '59.9259.5259.27' should presumably read '59.92 59.52 59.27'. Please fix the spacing.
  2. [§4.3 and Table 1] The Skywork-V2 adapter row in Table 1 is labeled 'hidden state', but the text in §4.3 says it uses the same hidden-state and normalized-score interface as CSPF. Please make the interface labeling consistent.
  3. [§5.3] The sentence 'the preferred representation differs by evaluation target' is exactly the kind of finding that creates a model-selection risk. If this is a post-hoc observation, it should be framed as such; if it is a testable hypothesis, it should be validated on a held-out portion of PPE.
  4. [General] No code or data availability statement is provided. Given the number of implementation details (readout extraction, projection layers, fusion architecture, hyperparameters), releasing code would substantially improve reproducibility.
  5. [Figure 3] The coincident traces for RM Ensemble are slightly offset, but the offset is small and the figure is hard to read. Consider using separate panels or a legend with explicit labels for each expert.

Circularity Check

0 steps flagged

No constructional circularity: CSPF is trained on external preference labels and evaluated on held-out benchmarks; no fitted constant is renamed as a prediction.

full rationale

The paper's central claim is an empirical result, not a derivation from inputs. Equations (1)-(11) define a trainable fusion module whose parameters are fit to LM-Arena pairwise human preferences; PPE is explicitly excluded from training and score-normalization statistics (Section 4.1), so the headline PPE off6=61.67 is an out-of-sample measurement rather than a restatement of an input. The signal readout and expert pool are design choices analyzed in Sections 5.3-5.4, but the paper never states a selection rule that uses PPE, and the main configuration is not the best PPE row in Table 3 (e.g., 0.875L all-token mean has PPE off6 61.70), so no specific reduction from PPE selection to the reported number can be exhibited. There are no self-citations, no imported uniqueness theorems, and no ansatz smuggled in via citation. The acknowledged limitations (e.g., no downstream best-of-N/DPO testing) are scope limits, not circular steps. The only caveat is test-set-selection hygiene around readout/pool choices, which is a correctness/robustness concern rather than constructional circularity.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 0 invented entities

No new physical entities are introduced. The main burden is a set of hand-chosen hyperparameters, architecture choices, and domain assumptions about the informativeness and alignability of reward-model hidden states.

free parameters (6)
  • λ_B (Barlow loss weight) = 1e-3
    Hand-chosen in Table A.2; controls shared-space constraint strength.
  • λ_S (SupCon loss weight) = 1e-4
    Hand-chosen in Table A.2; controls private-space constraint strength.
  • β (Barlow off-diagonal penalty) = 0.005
    Hand-chosen in Table A.2.
  • τ (SupCon temperature) = 0.10
    Hand-chosen in Table A.2.
  • Hidden-state readout = 0.75L, last-nonpad + response mean
    Selected after signal-representation analysis (Table 3); not justified by an independent validation-only protocol.
  • Active expert pool C = S+O+A
    Selected after expert-pool ladder analysis (Figure 4); main pool is the best on both LM-Arena and PPE, but selection independence from OOD is unclear.
axioms (4)
  • domain assumption Hidden states of frozen reward models contain preference-relevant complementary information beyond scalar scores.
    Central premise of Section 3.1; the entire method depends on this.
  • domain assumption LM-Arena crowd preferences are valid target-domain supervision and PPE human-preference slices measure non-verifiable preference OOD.
    Evaluation validity in Sections 4.1 and 4.4.
  • standard math Bradley-Terry pairwise preference model
    Used in Eq. (8) for the primary training objective.
  • domain assumption Barlow Twins and supervised contrastive losses impose shared/private structure that transfers OOD.
    Eqs. (9)-(10); supported only by ablation, not by a derivation.

pith-pipeline@v1.3.0-alltime-deepseek · 14098 in / 9370 out tokens · 87626 ms · 2026-08-01T09:06:07.413454+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of CSPF: A Constrained Shared-Private Fusion Method for Non-Verifiable Preference Evaluation." pith.science (2026). https://pith.science/paper/D3MBOADD

@misc{pith2026260720862,
  author       = {Pith},
  title        = {Pith review of: CSPF: A Constrained Shared-Private Fusion Method for Non-Verifiable Preference Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D3MBOADD}},
  note         = {Machine review of arXiv:2607.20862}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

At present, reliable evaluation of non-verifiable tasks remains challenging. Existing approaches often fail to adequately capture the diverse evaluative criteria underlying human preferences in such tasks. To this end, we propose Constrained Shared-Private Fusion (CSPF), a fusion method that treats heterogeneous frozen reward models as complementary evaluators and learns to integrate their hidden-state representations under pairwise human-preference supervision. CSPF decomposes each expert signal into shared and expert-private representations, encouraging cross-expert alignment while preserving complementary viewpoints. Across experiments on LM-Arena target-domain adaptation and PPE out-of-distribution preference evaluation, CSPF achieves the best performance on the primary metrics among the evaluated single-expert reward-model, scalar-score multi-expert, and rubric-judge baselines. Overall, CSPF suggests that fusing hidden-state representations provides a more expressive basis for preference assessment, offering a practical route toward integrated evaluative signals for non-verifiable preference tasks.

Figures

Figures reproduced from arXiv: 2607.20862 by Danli Wang, Hehao Zhang, Xinyuan Wang, Xuange Gao.

Figure 1
Figure 1. Figure 1: Overview of hidden-state evidence fusion from multiple reward experts for non-verifiable preference [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Architecture of CSPF. The left branch shows the per-expert signal path; the right branch illustrates [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Sample-level expert-contribution shares on the same 200 random PPE responses. RM Ensemble is [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Expert-pool ladder ablation for CSPF. +AM, +AC, +SA, and +U cumulatively add AceMath, AceCo [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

13 extracted references · 6 linked inside Pith

  1. [3]

    https://arxiv.org/abs/2509.21164v1

    Mixture of Thoughts: Learning to Aggregate What Experts Think, Not Just What They Say. https://arxiv.org/abs/2509.21164v1. Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E. Gonzalez, and Ion Sto- ica

    arXiv

  2. [4]

    Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Quentin Carbonneaux, Taco Cohen, and Gabriel Synnaeve

    How to Evaluate Reward Models for RLHF.International Conference on Learning Representations, 2025:18128–18163. Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Quentin Carbonneaux, Taco Cohen, and Gabriel Synnaeve

    2025

  3. [5]

    https://arxiv.org/abs/2410.02089v2

    RLEF: Grounding Code LLMs in Execu- tion Feedback with Reinforcement Learning. https://arxiv.org/abs/2410.02089v2. Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean M. Hendryx

    Pith/arXiv arXiv

  4. [6]

    InPro- ceedings of the 2024 Conference on Empiri- cal Methods in Natural Language Processing, pages 4334–4353, Miami, Florida, USA

    Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models. InPro- ceedings of the 2024 Conference on Empiri- cal Methods in Natural Language Processing, pages 4334–4353, Miami, Florida, USA. Asso- ciation for Computational Linguistics. Nathan Lambert, Valentina Pyatkin, Jacob Mor- rison, LJ Miranda, Bill Yuchen Lin, Khyat...

    2024

  5. [7]

    InFindings of the Association for Computa- tional Linguistics: NAACL 2025, pages 1755– 1797, Albuquerque, New Mexico

    RewardBench: Evalu- ating Reward Models for Language Modeling. InFindings of the Association for Computa- tional Linguistics: NAACL 2025, pages 1755– 1797, Albuquerque, New Mexico. Association for Computational Linguistics. Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, Yang Liu...

    Pith/arXiv arXiv 2025

  6. [8]

    https://arxiv.org/abs/2410.00847v2

    Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown. https://arxiv.org/abs/2410.00847v2. Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A. Smith, Hannaneh Ha- jishirzi, and Nathan Lambert

    Pith/arXiv arXiv

  7. [9]

    https://arxiv.org/abs/2506.01937v2

    Reward- Bench 2: Advancing Reward Model Evaluation. https://arxiv.org/abs/2506.01937v2. Duy Nguyen, Archiki Prasad, Elias Stengel-Eskin, and Mohit Bansal

    Pith/arXiv arXiv

  8. [10]

    InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 1043–1067, Miami, Florida, USA

    Off- setBias: Leveraging Debiased Data for Tun- ing Evaluators. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 1043–1067, Miami, Florida, USA. Asso- ciation for Computational Linguistics. Anna Rogers, Olga Kovaleva, and Anna Rumshisky

    2024

  9. [11]

    https://arxiv.org/abs/2402.03300v3

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. https://arxiv.org/abs/2402.03300v3. William F. Shen, Xinchi Qiu, Chenxi Whitehouse, Lisa Alazraki, Shashwat Goel, Francesco Bar- bieri, Timon Willi, Akhil Mathur, and Ilias Leontiadis

    Pith/arXiv arXiv

  10. [13]

    Depending on the experiment, we use their reward scores, hidden states, or both

    attribute / calibration Table A.1: Frozen scalar reward experts considered in our study. Depending on the experiment, we use their reward scores, hidden states, or both. Item Value Active expert pool S+O+A Hidden-state signal last-nonpad + response mean Layer specification0.75Lproportional layer Fusion structure shared/private encoders + structured hidden...

    2021

  11. [2024]

    https://arxiv.org/abs/2403.04132v1

    Chatbot Arena: An Open Plat- form for Evaluating LLMs by Human Prefer- ence. https://arxiv.org/abs/2403.04132v1. Jacob Eisenstein, Chirag Nagpal, Alekh Agar- wal, Ahmad Beirami, Alexander Nicholas D’Amour, Krishnamurthy Dj Dvijotham, Adam Fisch, Katherine A. Heller, Stephen Robert Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant

    Pith/arXiv arXiv

  12. [2025]

    https://arxiv.org/abs/2505.13388v3

    R3: Robust Rubric-Agnostic Reward Models. https://arxiv.org/abs/2505.13388v3. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica

    arXiv

  13. [2026]

    Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang

    Rethinking Rubric Gener- ation for Improving LLM Judge and Reward Modeling for Open-ended Tasks. Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. 2024a. Interpretable Preferences via Multi-Objective Reward Mod- eling and Mixture-of-Experts. InFindings of the Association for Computational Linguis- tics: EMNLP 2024, pages 10582–10592, Mi- a...

    2024