{"id":"7dfa2a2e-2d8d-4aad-9c80-25ac3231ce00","arxiv_id":"2507.04695","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A concept-bottleneck reward model with Expected Information Gain acquisition learns interpretable concepts faster under limited annotation budgets without hurting preference accuracy.","lead":"This paper builds reward models that explain their scores through human-readable concepts like helpfulness and correctness, and uses active learning to decide which concept labels to request. A generalist might read it because it targets a core problem in RLHF: making reward signals auditable while keeping annotation costs low.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation (8) defines EIG as the negative of BALD: since entropy is concave, E[H(p)]−H(E[p]) ≤ 0, so maximizing it selects low-information queries; the central claim is unsupported unless code uses the opposite formula.","rationale":"The reader's weakest assumption concerns the use of GPT-4o-judge concept annotations and the linear-combination preference labels; that is a valid external-validity concern. However, the more immediate and load-bearing problem is internal: Eq. (8) as written is the negative of BALD. The paper cites Houlsby et al. (2011), whose acquisition score is H(E[p]) − E[H(p)]; Eq. (8) reverses the order. Since H is concave, the expression in Eq. (8) is non-positive, so maximizing it is equivalent to minimizing mutual information between the queried concept label and the model parameters. That would be expected to slow, not accelerate, concept learning. Unless the implementation silently uses the standard ordering, the reported EIG results do not support the method's stated mechanism. This is not a rejection because the code is available and the issue may be only a typographical sign error; a straightforward code inspection and a rerun with the corrected formula would settle it. The reader's synthetic-label concern remains important for any human-alignment or interpretability claim, but the sign error is more fundamental to the central claim. Therefore the conditional verdict should be retained, with the additional condition that the authors verify and, if necessary, correct the EIG formula and reproduce Figure 2 using the standard BALD expression.","tokens_in":7854,"tokens_out":7034,"duration_ms":76193,"concrete_test":"Inspect the released code at github.com/sonialagunac/cb-rm-workshop: verify whether the acquisition score is computed as H(E[p]) − E[H(p)] (correct BALD) or as E[H(p)] − H(E[p]) (Eq. 8 as written). Then run a small toy example with a two-point posterior to confirm which ordering the implementation uses. If the code matches Eq. 8, rerun Figure 2 with the corrected BALD formula; if the EIG-versus-random concept-accuracy gap disappears or reverses, the paper's central empirical claim is an artifact of the sign error. If the code already uses the correct ordering, the paper should be revised to fix Eq. 8, and the empirical conclusions remain supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation (8) states AEIG(i,k) = E_{θ,ϕ}[H(p(s|θ,ϕ))] − H(E_{θ,ϕ}[p(s|θ,ϕ)]). The standard BALD acquisition score (Houlsby et al., 2011) is H(E[p]) − E[H(p)], which equals the mutual information between the concept label and the model parameters. Because entropy is concave, E[H(p)] ≤ H(E[p]); therefore Eq. (8) is the negative of that mutual information and is always ≤ 0. Maximizing Eq. (8) as written selects queries with the lowest expected information gain, not the highest. This is directly load-bearing because the paper's central claim is that the proposed EIG acquisition function accelerates concept learning; if the implementation follows Eq. (8) literally, the observed gains cannot be attributed to expected information gain, and the theoretical grounding of the method is inverted. The reader's concern about GPT-4o-generated concept labels and the linear-combination preference rule is real for external validity, but this sign error threatens the internal identity of the proposed method. The paper does release code and reports five-seed results, so the issue is checkable; it may be a typographical ordering error in the manuscript, but as written the method is not EIG.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces Concept Bottleneck Reward Models (CB-RM), in which a reward for a prompt-response pair is computed as w(x)^T c(x,y), where c is a probabilistic concept vector predicted by an encoder and w is a prompt-conditioned weight vector. Training requires binary concept labels (which response is better on each of ten concepts) plus preference labels. To reduce annotation cost the authors formalize an active-learning loop that queries concept labels and propose acquisition functions: random, concept variance, CwIS, and EIG as in Eq. (8). Experiments on UltraFeedback with GPT-4o-generated concept annotations and 5 seeds report that EIG improves concept accuracy fastest while preference accuracy stays comparable. They also include an appendix showing that with LLaMA-3 embeddings (potentially pretrained on UltraFeedback) active learning gives no gains, which they attribute to information leakage.","tokens_in":8132,"tokens_out":3683,"duration_ms":35743,"significance":"If the central empirical claim is correct, active concept acquisition is a sensible way to make reward models interpretable and cheaper to annotate, and the paper is among the first to formalize active learning in concept bottleneck models. Strengths include public code, five-seed results, explicit discussion of data leakage, and the choice of a pre-release encoder to avoid leakage in the main experiments. However, the significance is bounded by two issues: Eq. (8)'s EIG expression appears sign-inverted, and all labels come from a synthetic GPT-4o protocol in which preferences are defined as a linear combination of the concept labels. Both points bear directly on whether the reported gains can be attributed to expected information gain and on whether the preference-accuracy claim generalizes to human preferences.","major_comments":[{"comment":"Eq. (8) defines AEIG(i,k) = E_{θ,ϕ}[H(p(s|θ,ϕ))] − H(E_{θ,ϕ}[p(s|θ,ϕ)]). The standard BALD score is H(E[p]) − E[H(p)], which equals the mutual information between the label and the model parameters. Since entropy is concave, E[H(p)] ≤ H(E[p]) for any distribution, so the quantity in Eq. (8) is ≤ 0, and maximizing it selects the least informative queries. As written, the proposed acquisition function is the negative of expected information gain. This sign is load-bearing because the paper's central claim is that EIG accelerates concept learning: if the implementation follows Eq. (8) literally, the observed gains cannot be attributed to EIG; if the implementation uses the positive BALD score, the equation must be corrected. Because the code is released, this is checkable.","section":"§3, Eq. (8)"},{"comment":"The preference labels are constructed as a linear combination of the ten GPT-4o concept scores (Appendix B.2), and the reward model is itself a linear function of concept predictions (Eq. (2)). Under this protocol, 'preference accuracy' measures agreement with a synthetic rule derived from the same concept variables the model is trained to predict; it does not measure alignment with human preferences. The active-learning gains are therefore measured entirely against GPT-4o annotations, whose fidelity to human judgment is assumed rather than validated. The paper should state this limitation explicitly and temper the claim that CB-RM aligns with human preferences; ideally, a small human-labeled evaluation set or existing human preference judgments on UltraFeedback subsets would test the assumption.","section":"Appendix B.2 and Eq. (2)"},{"comment":"The EIG score is not operationalized. The manuscript never specifies how E_{θ,ϕ} is computed: the model has a probabilistic encoder fθ predicting Gaussian concept scores (Eq. (3)), but no posterior over θ,ϕ is defined, and the text does not say whether the expectation is approximated by Monte Carlo dropout, ensembles, or the predictive variance. It also does not define the likelihood p(s_k^i | x_i, y_i, y'_i, θ, ϕ) for the binary concept label given the Gaussian difference Δc. Without this, Eq. (8) cannot be reproduced and the reported EIG results cannot be audited.","section":"§3, Eq. (8) and Algorithm 1"}],"minor_comments":[{"comment":"The caption calls EIG an 'Activation Function (AF)'; this should be 'acquisition function'.","section":"Figure 1 caption"},{"comment":"The text says each concept was scored independently on a scale from 0 to 1, but the formalism defines binary labels s_k in {0,1}; clarify whether the labels are binary comparisons or continuous scores, and how the linear combination of these scores produces the final preference label.","section":"Appendix B.2"},{"comment":"The coefficients of the linear combination used to generate preference labels are not reported; this information is needed for reproducibility.","section":"Appendix B.2"},{"comment":"The CwIS acquisition function refers to intervening on the k-th concept by 'setting its logit to a fixed high/low value', but the fixed values are not specified.","section":"§3, Eq. (7)"},{"comment":"The abstract says EIG 'significantly accelerates' concept learning, but the paper reports only means and standard deviations over five seeds; adding a statistical comparison or effect sizes would support the word 'significantly'.","section":"Figure 2 and abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to be a workshop version; for a journal submission, the EIG sign issue and the synthetic-label protocol need to be resolved before the central claims can be evaluated. I did not find evidence of citation concerns; the critique of ArmoRM is reasonable and specific."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nKnow this paper as: a workshop paper with a genuinely new setup - active learning over (instance, concept) pairs for concept-bottleneck reward models - and a central acquisition function that looks wrong as written. Eq. (8) defines EIG as E[H(p)] - H(E[p]), which is the negative of BALD. Since entropy is concave, that expression is always <= 0, so maximizing it selects low-information queries. The paper's main claim is that this EIG accelerates concept learning. If the implementation follows the formula, the gains are not from expected information gain; if it uses the true BALD, the formula is a typo. Either way, the manuscript needs a fix and a code check. The reader didn't catch this; I think it's the biggest issue.\n\nWhat's genuinely good: the problem is real. Prior CBM work assumes full concept annotations at training; ArmoRM needs all concept scores. Making the reward model's bottleneck interpretable and deciding which concept labels to query is a sensible contribution. The paper runs five seeds, reports both concept and preference accuracy, and the concept accuracy curves for EIG look better than random. They also carefully avoid leakage by using LLaMA-2-7B embeddings, and Appendix C honestly shows that with LLaMA-3 embeddings (which likely saw UltraFeedback) active learning stops helping. That is a fair empirical observation.\n\nSoft spots besides the sign error: the preference labels are generated as a linear combination of the same ten concept scores used for training (Appendix B.2), and the reward model is linear in those concepts. So preference accuracy partly measures how well the model replicates a synthetic labeling rule, not human preference alignment. There is no human validation of the concept labels (GPT-4o judge is the ground truth). That's a limitation for external validity, not an internal inconsistency. Minor: EIG computation details (how the expectations over theta and phi are approximated) are unspecified.\n\nOverall: this is the kind of paper that a serious workshop referee could help fix. The core idea is worth a look, the empirical claim is plausible, and the main flaw is checkable by running the code. I'd send it to review with a request to verify Eq. (8) against the implementation and to add a non-circular preference benchmark. Not a desk reject, but not ready as is.","headline":"A genuinely new active-learning setup for concept-bottleneck reward models, but Eq. (8) defines EIG as the negative of BALD, and the central claim depends on checking the code.","tokens_in":8662,"tokens_out":3363,"would_cite":false,"duration_ms":36480,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A concept-bottleneck reward model with EIG-based active learning learns interpretable concepts faster without hurting preference accuracy.","keywords":["concept bottleneck models","reward modeling","RLHF","active learning","expected information gain","interpretability","preference learning","UltraFeedback"],"falsifier":"Re-run the 30-episode active learning loop on a sample of UltraFeedback with concept and preference labels collected from human annotators instead of GPT-4o; if EIG's concept-accuracy advantage over random selection shrinks or disappears, or preference accuracy no longer matches human choices, the reported gains are artifacts of the synthetic labeling rule.","tokens_in":7641,"feed_emoji":"🎯","tokens_out":6368,"duration_ms":62116,"temperature":0.7,"pith_summary":"The paper proposes Concept Bottleneck Reward Models (CB-RM), which replace opaque RLHF reward functions with a transparent structure: a model predicts scores for ten human-interpretable concepts per response, and a context-conditioned weighting turns those scores into a reward. Because labeling every concept for every pair is expensive, the paper formalizes an active learning loop that chooses which concept labels to query, and introduces an acquisition function based on Expected Information Gain (EIG). The central claim is that EIG-based acquisition significantly accelerates concept learning across training episodes compared with random selection, while preference accuracy stays comparable. The authors demonstrate this on UltraFeedback with an LLM-judge-generated concept ground truth, arguing that CB-RM is a step toward auditable, data-efficient reward models.","feed_headline":"Information-gain queries speed up reward concept learning","feed_subtitle":"A concept-bottleneck reward model learns auditable attributes with fewer labels and steady preference accuracy.","key_machinery":"The central object is the probabilistic concept bottleneck: an encoder $f_\\theta$ maps each prompt-response pair to a Gaussian over $K$ concept scores, and a context encoder $g_\\phi$ maps the prompt to a weight vector; the reward is the inner product $w^T c$. Because the difference between two responses' concept vectors is also Gaussian, acquisition functions can use both means and variances. The load-bearing identity is the EIG acquisition function, which approximates the expected reduction in entropy of the concept-label prediction and ranks candidate queries $(i,k)$; it is the mechanism that makes the label budget go further. Around it sits a FIFO replay buffer that keeps training on recent, informative samples.","core_discovery":"On its own terms, the paper's discovery is that treating concept annotations as the active-learning currency in a probabilistic concept-bottleneck reward model yields large gains in interpretability per label. The reward is $r(x,y)=g_\\phi(x)^T f_\\theta(x,y)$, where $f_\\theta$ outputs a Gaussian over concept scores and $g_\\phi$ outputs prompt-dependent concept weights, so the difference between two responses is Gaussian with known mean and variance. EIG selects the concept-query pair whose label is expected to reduce predictive entropy the most. In 30-episode runs on UltraFeedback, EIG reaches concept accuracies that random selection attains only later, while preference accuracy remains in the same range across methods. The paper also positions this as the first active-learning formalization for CBMs in preference learning, and shows that concept-level supervision adds little once the encoder already encodes the concepts (a leakage warning).","pith_inferences":["We infer that the EIG advantage is contingent on the judge's concept labels being informative and mutually non-redundant; an untested extension is swapping the GPT-4o judge for human annotators or for a different judge model and re-measuring the gap.","If the linear-combination reward is replaced by a learned monotone aggregation of the same concepts, preference accuracy might improve, but the paper's interpretability guarantee would weaken; this is a testable trade-off the paper leaves implicit.","An untested stress test is whether EIG still wins when the concept vocabulary is larger or noisier (e.g., 30 concepts or coarse ordinal labels), where information gain estimates from a Gaussian bottleneck may overestimate the value of a single label.","The leakage finding suggests a practical rule for practitioners: when embeddings already predict the concept labels with high accuracy at initialization, active learning is unnecessary, and the remaining signal is in the reward aggregation rather than in the concepts."],"forward_implications":["If the central claim holds, concept bottleneck reward models can be trained to the same concept accuracy with fewer annotated concept labels, lowering the cost of interpretable RLHF.","Preference accuracy being comparable across acquisition methods implies that interpretability can be added to reward models as a byproduct of good query selection, not at the price of reward quality.","Because CB-RM decomposes the reward into named concepts with prompt-dependent weights, failures in reward prediction can be traced back to specific attributes, enabling debugging and intervention.","The leakage analysis warns that reward-model evaluation on encoder representations from models trained after the preference data was released is confounded, so future comparisons should use pretraining-safe encoders or held-out response distributions."],"supporting_citations":[{"why":"Introduces Concept Bottleneck Models, the architecture CB-RM builds on to predict interpretable concepts before the final label.","marker":"Koh et al. (2020)"},{"why":"Provides the Bayesian active learning formulation and the Expected Information Gain criterion that the acquisition function approximates.","marker":"Houlsby et al. (2011)"},{"why":"Supplies the UltraFeedback dataset used for all experiments and preference-pair construction.","marker":"Cui et al. (2024)"},{"why":"Defines the BTL model that links reward differences to preference probabilities, grounding Equation (1).","marker":"Bradley & Terry (1952)"},{"why":"The prior interpretable reward-model work (ArmoRM) whose concept-annotation and leakage issues CB-RM explicitly addresses.","marker":"Wang et al. (2024)"},{"why":"Source of the CooP test-time intervention policy that inspires the CwIS acquisition function.","marker":"Chauhan et al. (2023)"},{"why":"Motivates the FIFO replay buffer strategy used to stabilise training in the active learning loop.","marker":"Dwaracherla et al. (2024)"}],"fun_headline_variants":["Active concept bottlenecks make reward models interpretable with fewer labels","Information-gain concept selection yields auditable reward models efficiently","EIG-guided concept labels speed up interpretable reward learning","Active concept queries make reward models transparent and sample-efficient","Selecting concepts by information gain yields interpretable rewards efficiently"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the GPT-4o judge's concept ratings are correct ground truth and that the final preference label is exactly a linear combination of those ten concept scores.","fun_headline_variants_meta":{"raw":{"variants":["Active concept bottlenecks make reward models interpretable with fewer labels","Information-gain concept selection yields auditable reward models efficiently","EIG-guided concept labels speed up interpretable reward learning","Active concept queries make reward models transparent and sample-efficient","Selecting concepts by information gain yields interpretable rewards efficiently"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001029,"raw_usage":{"total_tokens":4280,"prompt_tokens":832,"completion_tokens":3448,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":448,"completion_tokens_details":{"reasoning_tokens":3368}},"tokens_in":448,"tokens_out":3448,"duration_ms":28041,"temperature":1.0,"reasoning_tokens":3368,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:41:58.223929+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 30-episode active learning loop on a sample of UltraFeedback with concept and preference labels collected from human annotators instead of GPT-4o; if EIG's concept-accuracy advantage over random selection shrinks or disappears, or preference accuracy no longer matches human choices, the reported gains are artifacts of the synthetic labeling rule.","supporting_citations":[{"cited_title":", Nguyen, T","cited_arxiv_id":null,"evidence_quote":"Introduces Concept Bottleneck Models, the architecture CB-RM builds on to predict interpretable concepts before the final label."},{"cited_title":", Yuan, L","cited_arxiv_id":null,"evidence_quote":"Supplies the UltraFeedback dataset used for all experiments and preference-pair construction."},{"cited_title":"\\ Terry, M E","cited_arxiv_id":null,"evidence_quote":"Defines the BTL model that links reward differences to preference probabilities, grounding Equation (1)."},{"cited_title":", Xiong, W","cited_arxiv_id":null,"evidence_quote":"The prior interpretable reward-model work (ArmoRM) whose concept-annotation and leakage issues CB-RM explicitly addresses."},{"cited_title":", Tiwari, R","cited_arxiv_id":null,"evidence_quote":"Source of the CooP test-time intervention policy that inspires the CwIS acquisition function."},{"cited_title":", Asghari, S M","cited_arxiv_id":null,"evidence_quote":"Motivates the FIFO replay buffer strategy used to stabilise training in the active learning loop."}],"review_version":1}