Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Interpretable Reward Modeling with Active Concept Bottlenecks

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A concept-bottleneck reward model with EIG-based active learning learns interpretable concepts faster without hurting preference accuracy.

desk verdict A genuinely new active-learning setup for concept-bottleneck reward models, but Eq. (8) defines EIG as the negative of BALD, and the central claim depends on checking the code. read the letter →

arxiv 2507.04695 v2 pith:KLEHETQM submitted 2025-07-07 cs.LG

Sonia Laguna , Katarzyna Kobalczyk , Julia E. Vogt , Mihaela Van der Schaar This is my paper · ORCID
classification cs.LG
keywords conceptbottleneckmodelsrewardmodelingRLHFactivelearningexpectedinformationgaininterpretabilitypreferenceUltraFeedback
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Concept Bottleneck Reward Models (CB-RM), which replace opaque RLHF reward functions with a transparent structure: a model predicts scores for ten human-interpretable concepts per response, and a context-conditioned weighting turns those scores into a reward. Because labeling every concept for every pair is expensive, the paper formalizes an active learning loop that chooses which concept labels to query, and introduces an acquisition function based on Expected Information Gain (EIG). The central claim is that EIG-based acquisition significantly accelerates concept learning across training episodes compared with random selection, while preference accuracy stays comparable. The authors demonstrate this on UltraFeedback with an LLM-judge-generated concept ground truth, arguing that CB-RM is a step toward auditable, data-efficient reward models.

What carries the argument

The central object is the probabilistic concept bottleneck: an encoder $f_\theta$ maps each prompt-response pair to a Gaussian over $K$ concept scores, and a context encoder $g_\phi$ maps the prompt to a weight vector; the reward is the inner product $w^T c$. Because the difference between two responses' concept vectors is also Gaussian, acquisition functions can use both means and variances. The load-bearing identity is the EIG acquisition function, which approximates the expected reduction in entropy of the concept-label prediction and ranks candidate queries $(i,k)$; it is the mechanism that makes the label budget go further. Around it sits a FIFO replay buffer that keeps training on recent, informative samples.

What would settle it

Re-run the 30-episode active learning loop on a sample of UltraFeedback with concept and preference labels collected from human annotators instead of GPT-4o; if EIG's concept-accuracy advantage over random selection shrinks or disappears, or preference accuracy no longer matches human choices, the reported gains are artifacts of the synthetic labeling rule.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that treating concept annotations as the active-learning currency in a probabilistic concept-bottleneck reward model yields large gains in interpretability per label. The reward is $r(x,y)=g_\phi(x)^T f_\theta(x,y)$, where $f_\theta$ outputs a Gaussian over concept scores and $g_\phi$ outputs prompt-dependent concept weights, so the difference between two responses is Gaussian with known mean and variance. EIG selects the concept-query pair whose label is expected to reduce predictive entropy the most. In 30-episode runs on UltraFeedback, EIG reaches concept accuracies that random selection attains only later, while preference accuracy remains in the same range across methods. The paper also positions this as the first active-learning formalization for CBMs in preference learning, and shows that concept-level supervision adds little once the encoder already encodes the concepts (a leakage warning).

Load-bearing premise

The load-bearing premise is that the GPT-4o judge's concept ratings are correct ground truth and that the final preference label is exactly a linear combination of those ten concept scores.

Editorial extensions

If this is right

  • If the central claim holds, concept bottleneck reward models can be trained to the same concept accuracy with fewer annotated concept labels, lowering the cost of interpretable RLHF.
  • Preference accuracy being comparable across acquisition methods implies that interpretability can be added to reward models as a byproduct of good query selection, not at the price of reward quality.
  • Because CB-RM decomposes the reward into named concepts with prompt-dependent weights, failures in reward prediction can be traced back to specific attributes, enabling debugging and intervention.
  • The leakage analysis warns that reward-model evaluation on encoder representations from models trained after the preference data was released is confounded, so future comparisons should use pretraining-safe encoders or held-out response distributions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that the EIG advantage is contingent on the judge's concept labels being informative and mutually non-redundant; an untested extension is swapping the GPT-4o judge for human annotators or for a different judge model and re-measuring the gap.
  • If the linear-combination reward is replaced by a learned monotone aggregation of the same concepts, preference accuracy might improve, but the paper's interpretability guarantee would weaken; this is a testable trade-off the paper leaves implicit.
  • An untested stress test is whether EIG still wins when the concept vocabulary is larger or noisier (e.g., 30 concepts or coarse ordinal labels), where information gain estimates from a Gaussian bottleneck may overestimate the value of a single label.
  • The leakage finding suggests a practical rule for practitioners: when embeddings already predict the concept labels with high accuracy at initialization, active learning is unnecessary, and the remaining signal is in the reward aggregation rather than in the concepts.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript introduces Concept Bottleneck Reward Models (CB-RM), in which a reward for a prompt-response pair is computed as w(x)^T c(x,y), where c is a probabilistic concept vector predicted by an encoder and w is a prompt-conditioned weight vector. Training requires binary concept labels (which response is better on each of ten concepts) plus preference labels. To reduce annotation cost the authors formalize an active-learning loop that queries concept labels and propose acquisition functions: random, concept variance, CwIS, and EIG as in Eq. (8). Experiments on UltraFeedback with GPT-4o-generated concept annotations and 5 seeds report that EIG improves concept accuracy fastest while preference accuracy stays comparable. They also include an appendix showing that with LLaMA-3 embeddings (potentially pretrained on UltraFeedback) active learning gives no gains, which they attribute to information leakage.

Significance. If the central empirical claim is correct, active concept acquisition is a sensible way to make reward models interpretable and cheaper to annotate, and the paper is among the first to formalize active learning in concept bottleneck models. Strengths include public code, five-seed results, explicit discussion of data leakage, and the choice of a pre-release encoder to avoid leakage in the main experiments. However, the significance is bounded by two issues: Eq. (8)'s EIG expression appears sign-inverted, and all labels come from a synthetic GPT-4o protocol in which preferences are defined as a linear combination of the concept labels. Both points bear directly on whether the reported gains can be attributed to expected information gain and on whether the preference-accuracy claim generalizes to human preferences.

major comments (3)
  1. [§3, Eq. (8)] Eq. (8) defines AEIG(i,k) = E_{θ,ϕ}[H(p(s|θ,ϕ))] − H(E_{θ,ϕ}[p(s|θ,ϕ)]). The standard BALD score is H(E[p]) − E[H(p)], which equals the mutual information between the label and the model parameters. Since entropy is concave, E[H(p)] ≤ H(E[p]) for any distribution, so the quantity in Eq. (8) is ≤ 0, and maximizing it selects the least informative queries. As written, the proposed acquisition function is the negative of expected information gain. This sign is load-bearing because the paper's central claim is that EIG accelerates concept learning: if the implementation follows Eq. (8) literally, the observed gains cannot be attributed to EIG; if the implementation uses the positive BALD score, the equation must be corrected. Because the code is released, this is checkable.
  2. [Appendix B.2 and Eq. (2)] The preference labels are constructed as a linear combination of the ten GPT-4o concept scores (Appendix B.2), and the reward model is itself a linear function of concept predictions (Eq. (2)). Under this protocol, 'preference accuracy' measures agreement with a synthetic rule derived from the same concept variables the model is trained to predict; it does not measure alignment with human preferences. The active-learning gains are therefore measured entirely against GPT-4o annotations, whose fidelity to human judgment is assumed rather than validated. The paper should state this limitation explicitly and temper the claim that CB-RM aligns with human preferences; ideally, a small human-labeled evaluation set or existing human preference judgments on UltraFeedback subsets would test the assumption.
  3. [§3, Eq. (8) and Algorithm 1] The EIG score is not operationalized. The manuscript never specifies how E_{θ,ϕ} is computed: the model has a probabilistic encoder fθ predicting Gaussian concept scores (Eq. (3)), but no posterior over θ,ϕ is defined, and the text does not say whether the expectation is approximated by Monte Carlo dropout, ensembles, or the predictive variance. It also does not define the likelihood p(s_k^i | x_i, y_i, y'_i, θ, ϕ) for the binary concept label given the Gaussian difference Δc. Without this, Eq. (8) cannot be reproduced and the reported EIG results cannot be audited.
minor comments (5)
  1. [Figure 1 caption] The caption calls EIG an 'Activation Function (AF)'; this should be 'acquisition function'.
  2. [Appendix B.2] The text says each concept was scored independently on a scale from 0 to 1, but the formalism defines binary labels s_k in {0,1}; clarify whether the labels are binary comparisons or continuous scores, and how the linear combination of these scores produces the final preference label.
  3. [Appendix B.2] The coefficients of the linear combination used to generate preference labels are not reported; this information is needed for reproducibility.
  4. [§3, Eq. (7)] The CwIS acquisition function refers to intervening on the k-th concept by 'setting its logit to a fixed high/low value', but the fixed values are not specified.
  5. [Figure 2 and abstract] The abstract says EIG 'significantly accelerates' concept learning, but the paper reports only means and standard deviations over five seeds; adding a statistical comparison or effect sizes would support the word 'significantly'.

Circularity Check

1 steps flagged · score 4.0 of 10

No load-bearing self-citation; the main definitional problem is Eq. (8), which defines EIG as the negative of standard BALD, and the preference labels are synthesized from the same concepts the model predicts.

  1. self definitional [Section 3, Eq. (8)]
    "EIG maximizes the expected reduction in uncertainty about the model's predictions after observing a concept label, which can be approximated by computing the difference between the expected entropy and the entropy of the expected prediction after sampling (Houlsby et al., 2011). Concretely, we have: AEIG(i,k) = E_{θ,ϕ}[H(p(s_i^k | x_i, y_i, y'_i, θ,ϕ))] − H(E_{θ,ϕ}[p(s_i^k | x_i, y_i, y'_i, θ,ϕ)]). (8)"

    Entropy is concave, so Jensen's inequality gives H(E[p]) ≥ E[H(p)]; hence Eq. (8) is always ≤ 0 and equals the negative of the standard BALD mutual-information acquisition score cited from Houlsby et al. Maximizing Eq. (8) therefore selects concept-query pairs with the smallest expected information gain, not the largest. The paper's central claim that EIG 'significantly accelerates concept learning without compromising preference accuracy' is attributed to a quantity whose written definition is the opposite of expected information gain. Unless the released code evaluates the reversed sign (making the displayed formula a typographical inversion), the empirical comparison does not test EIG; the claimed benefit is definitionally detached from the equation as presented.

full rationale

The derivation chain is mostly self-contained: the model is a standard Concept Bottleneck Model built on external work (Koh et al., 2020), the active-learning comparison is empirical on held-out data with five seeds, and the self-citations (Vandenhirtz et al., 2024; Laguna et al., 2024; Makonnen et al., 2025; Kobalczyk & van der Schaar, 2025) are background or interpretative, not load-bearing. No fitted constant is reused as a prediction, and no uniqueness theorem is imported from the authors' prior work. The main circularity-adjacent problem is Eq. (8), where the formula presented as Expected Information Gain is the negative of the standard BALD score; as written, the acquisition function selects low-information queries, so the central claim is not supported by the paper's own definition. Separately, Appendix B.2 states that final preference labels are generated as a linear combination of the same ten concepts the model predicts, so 'preference accuracy' measures consistency with a synthetic linear labeling rule rather than independent human preference. That limits external validity, but it is not a formal circularity because the model still must learn concepts from text and generalize to held-out data. Score 4 reflects one definitional inversion in the core acquisition function while the remainder of the empirical comparison is checkable and not fitted to the target result.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on several modeling choices: the BTL preference model, a linear reward decomposition into concepts, Gaussian concept uncertainty, and, most importantly, the use of GPT-4o as the source of all concept and preference labels, with preference labels defined as a linear combination of the concept labels. None of these are validated against human annotators, and the linear coefficients are not reported.

free parameters (3)
  • CwIS weight lambda = 0.1
    Hand-set in Eq. 7 to balance influence and variance; affects the CwIS baseline, not EIG.
  • Preference label linear combination coefficients = unspecified
    Appendix B.2 states preference labels are generated as a linear combination of the ten concepts, but the coefficients are not given, making the preference target an unverifiable design choice.
  • Concept set and count (K=10) = helpfulness, correctness, coherence, complexity, verbosity, instruction following, truthfulness, honesty, safety…
    Chosen by the authors for relevance; the choice is not justified empirically and conditions all results.
assumptions (5)
  • standard math Bradley-Terry model for human preferences (Eq. 1)
    Assumed ground-truth human preference distribution.
  • domain assumption Reward is a linear function of concept scores: r = w^T c (Eq. 2)
    Assumes concepts are sufficient and the combination is linear; not justified from data.
  • domain assumption Concept predictions are independent Gaussians (Eq. 3-4)
    Postulated to enable the EIG acquisition; no calibration evidence.
  • ad hoc to paper LLM judge (GPT-4o) concept annotations are ground truth
    All concept and preference labels come from a GPT-4o judge rather than human annotators; the central evaluation depends on this proxy.
  • ad hoc to paper Preference label is a linear combination of concept labels
    Appendix B.2 states preference labels are generated by a linear combination of the 10 concepts, coupling the two objectives.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Interpretable Reward Modeling with Active Concept Bottlenecks." pith.science (2026). https://pith.science/paper/KLEHETQM

@misc{pith2026250704695,
  author       = {Pith},
  title        = {Pith review of: Interpretable Reward Modeling with Active Concept Bottlenecks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KLEHETQM}},
  note         = {Machine review of arXiv:2507.04695}
}
read the original abstract

We introduce Concept Bottleneck Reward Models (CB-RM), a reward modeling framework that enables interpretable preference learning through selective concept annotation. Unlike standard RLHF methods that rely on opaque reward functions, CB-RM decomposes reward prediction into human-interpretable concepts. To make this framework efficient in low-supervision settings, we formalize an active learning strategy that dynamically acquires the most informative concept labels. We propose an acquisition function based on Expected Information Gain and show that it significantly accelerates concept learning without compromising preference accuracy. Evaluated on the UltraFeedback dataset, our method outperforms baselines in interpretability and sample efficiency, marking a step towards more transparent, auditable, and human-aligned reward models.

Figures

Figures reproduced from arXiv: 2507.04695 by the authors.

Figure 1
Figure 1. Overview of CB-RM with Active Learning. In CB-RM, each prompt-response pair is encoded by an LLM encoder and an MLP to predict Gaussian-distributed concept representations ∆c, to generate a reward prediction using context-conditioned weights w from gϕ(x). During Active Learning, concept-label pairs are selectively acquired using an Activation Function (AF), the best performing being EIG, to maximize concept learning… view at source ↗
Figure 2
Figure 2. Top: Concept and preference accuracy in the EIG and random acquisition strategies. Bottom: Concept and preference accuracy in all studied acquisition strategies. Shaded areas denote standard deviation across 5 seeds. 6. Discussion and Conclusion Although preference accuracy remains comparable across methods, models trained with EIG are notably more robust and interpretable in terms of concept performance ( [PITH_FU… view at source ↗
Figure 3
Figure 3. Concept and preference accuracy in all studied acquisition strategies using embeddings generated from Llama3-8B. Shaded areas denote standard deviation across 5 seeds. 7 [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CraftAlign: Feature-Grounded Evaluation and Revision Guidance for AI Stories

    cs.AI 2026-08 conditional novelty 6.0 of 10

    CraftAlign uses 304 writing features and an energy model to choose which aspects of an AI story to revise, and its guidance improves perceived human-likeness in a small human study.

Reference graph

Works this paper leans on

21 extracted references · 17 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    APACrefauthors \ 2023

    meta2023llama2 APACrefauthors AI, M. APACrefauthors \ 2023 . LLaMA 2 7B model. Llama 2 7b model. https://huggingface.co/meta-llama/Llama-2-7b

    work page 2023

  3. [3]

    , Jones, A

    bai2022training APACrefauthors Bai, Y. , Jones, A. , Ndousse, K. , Askell, A. , Chen, A. , DasSarma, N. others APACrefauthors \ 2022 . Training a helpful and harmless assistant with reinforcement learning from human feedback Training a helpful and harmless assistant with reinforcement learning from human feedback . arXiv preprint arXiv:2204.05862

    arXiv 2022

  4. [4]

    \ Terry, M E

    bradley1952rank APACrefauthors Bradley, R A. \ Terry, M E. APACrefauthors \ 1952 . Rank analysis of incomplete block designs: I. The method of paired comparisons Rank analysis of incomplete block designs: I. the method of paired comparisons . Biometrika 39 3/4 324--345

    work page 1952

  5. [5]

    , Davies, X

    casper2023open APACrefauthors Casper, S. , Davies, X. , Shi, C. , Gilbert, T K. , Scheurer, J. \ . APACrefauthors \ 2023 . Open problems and fundamental limitations of reinforcement learning from human feedback Open problems and fundamental limitations of reinforcement learning from human feedback . Transactions on Machine Learning Research

    work page 2023

  6. [6]

    , Tiwari, R

    chauhan2023interactive APACrefauthors Chauhan, K. , Tiwari, R. , Freyberg, J. , Shenoy, P. \ Dvijotham, K. APACrefauthors \ 2023 . Interactive concept bottleneck models Interactive concept bottleneck models . Proceedings of the AAAI Conference on Artificial Intelligence Proceedings of the aaai conference on artificial intelligence \ ( 37, \ 5948--5955)

    work page 2023

  7. [7]

    , Leike, J

    christiano2017deep APACrefauthors Christiano, P F. , Leike, J. , Brown, T. , Martic, M. , Legg, S. \ Amodei, D. APACrefauthors \ 2017 . Deep reinforcement learning from human preferences Deep reinforcement learning from human preferences . Advances in neural information processing systems 30

    work page 2017

  8. [8]

    , Yuan, L

    cui2024ultrafeedback APACrefauthors Cui, G. , Yuan, L. , Ding, N. , Yao, G. \ . APACrefauthors \ 2024 . ULTRAFEEDBACK: boosting language models with scaled AI feedback Ultrafeedback: boosting language models with scaled ai feedback . Proceedings of the 41st International Conference on Machine Learning Proceedings of the 41st international conference on ma...

    work page 2024

Show all 21 references
  1. [9]

    \ Kim, B

    doshi2017towards APACrefauthors Doshi-Velez, F. \ Kim, B. APACrefauthors \ 2017 . Towards a rigorous science of interpretable machine learning Towards a rigorous science of interpretable machine learning . arXiv preprint arXiv:1702.08608

    2017 arXiv

  2. [10]

    , Asghari, S M

    dwaracherla2024efficient APACrefauthors Dwaracherla, V. , Asghari, S M. , Hao, B. \ Van Roy, B. APACrefauthors \ 2024 . Efficient exploration for LLMs Efficient exploration for llms . Proceedings of the 41st International Conference on Machine Learning Proceedings of the 41st ...

    2024

  3. [11]

    , Husz \'a r, F

    houlsby2011bayesian APACrefauthors Houlsby, N. , Husz \'a r, F. , Ghahramani, Z. \ Lengyel, M. APACrefauthors \ 2011 . Bayesian active learning for classification and preference learning Bayesian active learning for classification and preference learning . arXiv preprint arXiv...

    2011 arXiv

  4. [12]

    \ van der Schaar, M

    kobalczyk2025preference APACrefauthors Kobalczyk, K. \ van der Schaar, M. APACrefauthors \ 2025 . Preference Learning for AI Alignment: a Causal Perspective Preference learning for AI alignment: a causal perspective . Forty-second International Conference on Machine Learning. ...

    2025

  5. [13]

    , Nguyen, T

    koh2020concept APACrefauthors Koh, P W. , Nguyen, T. , Tang, Y S. , Mussmann, S. , Pierson, E. , Kim, B. \ Liang, P. APACrefauthors \ 2020 . Concept bottleneck models Concept bottleneck models . Proceedings of the 37th International Conference on Machine Learning Proceedings o...

    2020

  6. [14]

    , Marcinkevi c s, R

    laguna2024beyond APACrefauthors Laguna, S. , Marcinkevi c s, R. , Vandenhirtz, M. \ Vogt, J. APACrefauthors \ 2024 . Beyond concept bottleneck models: How to make black boxes intervenable? Beyond concept bottleneck models: How to make black boxes intervenable? Advances in neur...

    2024

  7. [15]

    , Vandenhirtz, M

    makonnen2025measuring APACrefauthors Makonnen, M. , Vandenhirtz, M. , Laguna, S. \ Vogt, J E. APACrefauthors \ 2025 . Measuring leakage in concept-based methods: An information theoretic approach Measuring leakage in concept-based methods: An information theoretic approach . I...

    2025

  8. [16]

    , Tigas, P

    melodeep APACrefauthors Melo, L C. , Tigas, P. , Abate, A. \ Gal, Y. APACrefauthors \ 2024 . Deep Bayesian Active Learning for Preference Modeling in Large Language Models Deep bayesian active learning for preference modeling in large language models . The Thirty-eighth Annual...

    2024

  9. [17]

    ouyang2022training APACrefauthors Ouyang, L. , Wu, J. , Jiang, X. , Almeida, D. , Wainwright, C L. , Mishkin, P. others APACrefauthors \ 2022 . Training language models to follow instructions with human feedback Training language models to follow instructions with human feedba...

    2022

  10. [18]

    , Tong, M

    sharma2024towards APACrefauthors Sharma, M. , Tong, M. , Korbak, T. , Duvenaud, D. , Askell, A. \ . APACrefauthors \ 2024 . TOWARDS UNDERSTANDING SYCOPHANCY IN LANGUAGE MODELS Towards understanding sycophancy in language models . 12th International Conference on Learning Repre...

    2024

  11. [19]

    shin2023closer APACrefauthors Shin, S. , Jo, Y. , Ahn, S. \ Lee, N. APACrefauthors \ 2023 . A closer look at the intervention procedure of concept bottleneck models A closer look at the intervention procedure of concept bottleneck models . International Conference on Machine L...

    2023

  12. [20]

    , Laguna, S

    vandenhirtz2024stochastic APACrefauthors Vandenhirtz, M. , Laguna, S. , Marcinkevi c s, R. \ Vogt, J. APACrefauthors \ 2024 . Stochastic concept bottleneck models Stochastic concept bottleneck models . Advances in Neural Information Processing Systems 37 51787--51810

    2024

  13. [21]

    , Xiong, W

    wang2024interpretable APACrefauthors Wang, H. , Xiong, W. , Xie, T. , Zhao, H. \ Zhang, T. APACrefauthors \ 2024 . Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts Interpretable preferences via multi-objective reward modeling and mixture-of-...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.