{"id":"6bec3a91-abab-4c96-8812-132b3770088a","arxiv_id":"2505.20088","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A concept-discovery plus hierarchical regression pipeline explains human, LLM-judge, and reward-model preferences at local and global levels across eight domains.","lead":"This paper builds an automated pipeline that uses an LLM to discover concepts that separate preferred from rejected responses, then fits a white-box hierarchical regression to show which concepts matter for humans, LLM judges, and reward models across eight domains. The explanations are stress-tested by using them to guide response generation and to break judge ties, yielding modest but consistent gains.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Coverage of the fixed concept vocabulary for non-human mechanisms is untested; explanations may be incomplete or misleading.","rationale":"The reader's weakest assumption—concept vocabulary coverage—is the most load-bearing concern because it challenges the central claim at its foundation: if a mechanism's true drivers are absent from the dictionary, no amount of regression or explanation analysis can recover them, and the resulting explanations may attribute decisions to correlated proxies. This concern is supported by the paper's own admission in Appendix C.1.2 that the planned discovery ablation using a different preference source was not completed, and by the manual injection of ten fixed concepts, which further qualifies the 'fully automated' claim. The application-driven evaluations (Judge Hack and Tie Break) provide strong evidence that the discovered concepts are useful and influential, but they do not establish completeness: they show that the identified concepts matter, not that they are the only or primary drivers. A completed mechanism-specific discovery ablation would directly test whether the vocabulary is sufficient, either by showing high overlap (supporting the current explanations) or by revealing new high-impact concepts (requiring qualification). Other concerns, such as same-model circularity for Gemini-Pro and lack of error bars, are real but secondary: they affect interpretation of specific results rather than the central claim's general validity. The HMDR lift derivation is an approximation, but the application tests rely on weight rankings rather than exact lift values, so it is less load-bearing. Given the credible but addressable coverage gap, the reader's CONDITIONAL verdict remains appropriate, and this stress-test does not change it.","tokens_in":38569,"tokens_out":8173,"duration_ms":92602,"concrete_test":"Run the completed mechanism-specific discovery ablation: for each explained mechanism, apply the §3.1 pipeline to a sample of its own chosen/rejected triplets (e.g., 300 batches per mechanism), merge newly discovered non-duplicate concepts with the existing 624, retrain HMDR, and compare (a) held-out prediction accuracy and (b) top-lift concept rankings against the original vocabulary. If accuracy rises by more than ~2 points or top-10 explanations change materially, the fixed human-derived vocabulary is incomplete and the central claim requires qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that explanations capture what drives each mechanism depends on the concept vocabulary containing the concepts that mechanism actually uses. The vocabulary is discovered in §3.1 from triplets labeled by human preferences (plus ten manually fixed concepts, §C.1.2), then reused for all twelve mechanisms, including reward models and nine LLM judges. Appendix C.1.2 states the planned ablation using Gemini-1.5-Flash preferences for discovery was not completed due to budget, so there is no evidence that mechanism-specific concepts would be discovered. If a mechanism relies on a concept outside the 624-term dictionary, the HMDR regression can only attribute its decisions to correlated proxies, yielding an explanation that is incomplete or actively misleading. The paper offers no coverage test: prediction accuracy (≈66% for humans, ≈80% for judges) is not compared to a concept-rich ceiling, and the lift decomposition (Appendix C.3) is computed over this fixed, possibly under-specified feature set. Because the abstract's 'fully automated' claim is also weakened by the manually injected fixed concepts, the missing coverage evidence is the load-bearing gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a four-stage pipeline for concept-based explainability of preference mechanisms (human preferences, LLM-as-a-Judge models, and reward models) across eight domains. An LLM (Gemini-1.5-Pro) discovers candidate concepts from preference triplets, represents each triplet as a concept vector (Comp-rep or Score-rep), and a white-box Hierarchical Multi-Domain Regression (HMDR) model learns shared and domain-specific concept weights. The authors evaluate the pipeline by predicting preferences for twelve mechanisms, validating LLM concept annotations with a human study, and running two application-driven tests: Judge Hack (generating responses guided by top-ranked concepts improves the explained judge's preference) and Tie Break (re-prompting judges with human-explanation concepts improves agreement with human preferences). The paper claims the method is fully automated and that the explanations capture what actually drives each mechanism, with prediction performance comparable to the strongest few-shot LLM ensemble baselines.","tokens_in":38787,"tokens_out":6925,"duration_ms":73495,"significance":"If the identified gaps are addressed, this would be a substantial contribution to NLP interpretability: it scales concept-based explainability to multiple domains and mechanisms using LLMs, introduces a white-box hierarchical regression model that decomposes effects into shared and domain-specific components, and proposes application-driven protocols for evaluating explanation quality. The scale of the study (twelve mechanisms, eight domains, 624 concepts) and the release of code are strong assets. The demonstration that explanation-guided generation is preferred by the explained judges, and that concept-guided re-prompting can improve judge-human agreement, are valuable and credible directions. However, the central claim that explanations capture what drives each mechanism is currently under-supported by the concept-coverage evidence and by the lack of statistical rigor in several head-to-head comparisons. These issues are fixable within the manuscript's scope, but they materially affect the strength of the claims as written.","major_comments":[{"comment":"The abstract and Section 1 claim the method is 'fully automated,' but Appendix C.1.2 states that ten concepts were 'manually extracted' and designated as fixed concepts, and that these fixed concepts are explicitly injected into the discovery prompt in 50% of batches. More importantly, concept discovery is performed only on triplets labeled by human preferences in the main setup, and the planned ablation using Gemini-1.5-Flash labels 'was not completed due to budget.' Because the same 624-concept vocabulary is then reused to explain all twelve mechanisms, there is no evidence that the vocabulary covers concepts actually used by the nine LLM judges and two reward models. If a mechanism relies on a concept outside this dictionary, the HMDR regression can only attribute its decisions to correlated proxies, which would make the explanation incomplete or actively misleading. The authors should provide a coverage test (e.g., discover concepts from a held-out mechanism's labels on a subset and compare, or at least analyze whether prediction errors concentrate on examples where the relevant concepts are out of vocabulary) and should either remove the 'fully automated' claim or qualify it in light of the manual fixed concepts.","section":"Section 3.1 and Appendix C.1.2"},{"comment":"The headline accuracy comparisons are reported without error bars, confidence intervals, or significance tests. For human preferences, Ours (Comp-rep) achieves 66.0% and the strongest LLM-as-a-Judge baseline (the Gemini-Flash 10-shot ensemble) also achieves 66.0%, and several rows in Table 1 differ by less than one percentage point. Without variance estimates over the 25 in-domain splits and 5 out-of-domain seeds, the claim that the method is 'comparable to or better than' black-box baselines is not statistically supported. Please report standard deviations and run paired significance tests for the primary head-to-head comparisons, at least for the human-preference row and the mean rows in Tables 1 and 4.","section":"Figure 3 and Table 1"},{"comment":"The Judge Hack results are tested only against the 50% chance level (with Bonferroni correction), not against the random-concept control. For the QRM judge with Gemini-Flash generator, the explanation-guided win rate is 50.1%, barely above chance, while the random-concept control is 36.0%. A paired statistical comparison between explanation-guided and random-guided responses across the 4,800 comparisons (or per judge-generator pair) is needed to support the claim that explanations outperform random concepts 'by a much larger margin.' Adding confidence intervals for the win rates would also help the reader assess the reliability of the effect.","section":"Section 5.2, Table 2"},{"comment":"The text claims a 'consistent and meaningful trend' with the accuracy ranking 'random ≤ LaaJ ≤ human ≤ differences' across judges, but Table 3 contains violations: for Gemini-1.5-Flash zero-shot, Random Concepts (+1.8) outperforms the LaaJ's own explanation (+0.0), and for GPT-4o-mini with 10-shot, Random Concepts (+5.2) outperforms the GPT-4o-m explanation (+2.1). These differences are also reported without significance testing. The monotonic-ranking claim is therefore stronger than the data support. Please recompute with paired significance tests and temper the conclusion accordingly, or explain why these violations are consistent with the proposed narrative.","section":"Section 5.2, Table 3"},{"comment":"The lift decomposition derives E[lift_j] ≈ 0.5 (b_j + s_j) by assuming E[σ(z)] = 0.5 from the symmetric augmentation and by asserting that the second-order Taylor remainder is negligible because |Δz_j| < 1/4. The bound on |Δz_j| is not derived or empirically verified for the trained models, and the expectation requires that the test distribution be symmetric in the same way as the augmented training distribution. Please provide empirical verification of the bound and of the symmetry assumption on the actual test sets, or present the result as an approximation with these caveats clearly stated.","section":"Appendix C.3"}],"minor_comments":[{"comment":"The abstract states that the method 'outperforms baselines,' while Section 5.1 reports accuracy 'comparable to the strongest baseline' (66.0% vs 66.0%). Please revise the abstract to say 'comparable to or better than' to avoid overclaiming.","section":"Abstract and Figure 3"},{"comment":"The sentence 'we find that CoT prompting degrades LaaJ performance, whereas few-shot prompting improves it, consistent with Calderon et al. (2025)' appears twice consecutively in the same paragraph. Please remove the duplicate.","section":"Section 5.1"},{"comment":"The contribution list contains the phrase 'an fully automated method'; this should be 'a fully automated method.'","section":"Section 1"},{"comment":"The caption and text report that there are 4,800 total comparisons, but the per-cell number of comparisons used for each judge-generator pair is not broken out. Please add this information so that the win rates can be interpreted with appropriate confidence.","section":"Table 2"},{"comment":"The claim that 'our method is the first complete pipeline for preferences explainability' is strong given prior work on concept-based regression and LLM-assisted concept discovery; please soften the claim or situate it more carefully against the baselines discussed in Section 2.","section":"Section 4.2"},{"comment":"The baseline row labeled '0.0' is not explicitly defined in the table; the text indicates it is the accuracy without resolving ties, but this should be stated in the caption for clarity.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript tackles an important problem and the application-driven evaluation is a genuine strength. The main risk is the concept-coverage gap: the 'fully automated' claim is undercut by the manually injected fixed concepts, and the planned mechanism-specific discovery ablation was not completed, leaving the cross-mechanism explanations without a coverage guarantee. A second concern is the circularity for the Gemini-Pro row, where the concept annotator and the explained mechanism are the same model; a control with a different annotator LLM would strengthen that row. The lack of error bars and significance tests in several key comparisons is also likely to attract reviewer criticism. These issues are addressable and do not require re-doing the entire study, hence major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a solid contribution to concept-based explainability of preferences. The main novelty is the end-to-end pipeline: automatic concept discovery with an LLM, concept-vector representations, and the hierarchical multi-domain regression (HMDR) that separates shared and domain-specific effects. That integration hasn't appeared before, and the design is sensible. The authors also deserve credit for the evaluation strategy—Judge Hack and Tie Break are clever application-driven tests showing the explanations carry real signal, not just correlation. The human annotation validation and the released code add credibility.\n\nWhere I'd push back is the coverage of the concept vocabulary. The concepts are discovered from human-preference triplets (plus ten manually fixed concepts), then reused to explain all twelve mechanisms, including reward models and LLM judges. Appendix C.1.2 openly states the planned ablation with Gemini-Flash preferences wasn't completed, so there's no evidence that mechanism-specific concepts would be discovered. If a mechanism relies on a concept outside the 624-term dictionary, the regression can only attribute its decisions to correlated proxies, which could mislead. That's a load-bearing gap for the central claim that the explanations capture what drives each mechanism. I don't think it's fatal—the dictionary is large and the application experiments point in the right direction—but it needs a coverage test or at least a clear caveat.\n\nTwo smaller issues: key accuracy comparisons in Figure 3 and Table 1 lack error bars, and the Gemini-Pro mechanism is explained using features generated by Gemini-Pro itself, which can inflate apparent fidelity. Both are addressable.\n\nOverall, this is a well-executed paper that delivers a reusable method and honest evaluations. It deserves a serious referee and likely publication after the coverage question is addressed.","headline":"A genuinely useful concept-based explainability pipeline for preference mechanisms, with a real but addressable gap: the concept dictionary is built from human preferences and untested for the other eleven mechanisms.","tokens_in":39273,"tokens_out":2600,"would_cite":true,"duration_ms":47470,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-discovered concepts reveal what drives preference judgments.","keywords":["concept-based explainability","LLM-as-a-Judge","reward models","preference learning","multi-domain learning","white-box model","hierarchical regression","LLM alignment"],"falsifier":"Train the same HMDR pipeline with concepts discovered separately from each explained mechanism's own preference labels and compare accuracy and top concepts against the human-derived 624-concept dictionary; if the mechanism-specific dictionary materially improves prediction or surfaces top concepts absent from the shared set, then the paper's claim that the explanations capture that mechanism's drivers is falsified.","tokens_in":38356,"feed_emoji":"🧠","tokens_out":7376,"duration_ms":73459,"temperature":0.7,"pith_summary":"Human preferences, LLM-as-a-Judge, and reward models all select between responses, but what actually drives their choices is largely opaque. This paper tries to establish that a fully automated pipeline can make those drivers visible: an LLM discovers human-interpretable concepts that separate chosen from rejected responses, represents each query-response pair as a concept vector, and trains a white-box hierarchical regression that attributes preference decisions to shared and domain-specific concept weights. The authors claim these explanations are usable: when LLMs generate responses guided by the top concepts of a judge, that judge prefers them over unguided or randomly guided responses, and re-prompting judges with concepts important to humans resolves tie cases in line with human preferences. If true, this gives alignment and evaluation work a scalable way to inspect and steer the mechanisms that decide what counts as a good response.","feed_headline":"LLM-discovered concepts reveal what drives preference judgments","feed_subtitle":"Same concepts that predict preferences can steer LLM outputs and resolve judge ties toward human judgment.","key_machinery":"The load-bearing object is the Hierarchical Multi-Domain Regression (HMDR) model, a white-box logistic regression whose per-domain weight vector is the sum $\\beta^{(d)} = b + s^{(d)}$ of a shared vector $b$ and a domain-specific deviation $s^{(d)}$. Its objective adds the standard domain-specific log-loss and an extra shared loss that forces $b$ alone to be predictive, with $\\ell^1$ penalties on both components for sparsity. This decomposition is what lets a single model explain twelve mechanisms across eight domains while still generalizing to unseen domains using only the shared weights; explanation strength is the lift $0.5(b_j+s^{(d)}_j)$ for concept $j$ in domain $d$.","core_discovery":"The central discovery is that concept-based explanations of preferences can be produced end-to-end without manual concept curation, and that those explanations capture the concepts that actually influence each mechanism's choices. The pipeline discovers concepts from preference triplets by batching queries and asking an LLM to propose concepts that distinguish chosen from rejected responses, deduplicates them into a 624-concept dictionary shared across eight domains, and encodes each triplet as a vector (comparison-based Comp-rep or score-difference Score-rep). A white-box Hierarchical Multi-Domain Regression model then predicts each mechanism's choices, decomposing concept weights into a shared component and domain-specific deviations; explanation importance is read off as the expected probability lift, about $50(b_j+s^{(d)}_j)$. The paper reports that this explainable model matches the strongest few-shot LLM ensemble at 66.0% accuracy on human preferences, and that in application-driven tests explanations guide generation to win over judges and improve tie resolution with humans by up to 10.8 points.","pith_inferences":["Beyond the paper: the shared-versus-specific weight decomposition gives a compact signature of each mechanism, so one could compare a new reward model to human preferences by the distance between their weight vectors without collecting new explanations.","Beyond the paper: since the dictionary is built from human-labeled triplets, a cheap coverage test—ask each judge to name its own concepts and check overlap with the 624-concept set—would determine whether judge-specific drivers are missing; the paper does not run this test.","Beyond the paper: the Tie Break result suggests concept-guided re-prompting could serve as a general position-bias correction in LLM evaluation, applied not only to exact ties but to near-ties or unstable predictions.","Beyond the paper: because Score-rep gives finer-grained concept values, HMDR weights trained on it could be used as features to predict which mechanisms will agree with human raters on a new domain, a testable extension of the authors' analysis."],"forward_implications":["If a judge's top concepts are fed into a generator, the judge prefers the resulting responses over vanilla ones, with win-rate gains up to +20.8 points over randomly chosen concepts for Gemini-Pro.","Re-prompting a judge on tie cases with concepts humans care about raises agreement with human preferences by up to 10.8 points, with the human-minus-judge weight difference working best.","The white-box model matches a strong eight-prompt few-shot Gemini-Flash ensemble at 66.0% accuracy on human preferences while remaining interpretable.","In leave-one-domain-out testing, the shared weights alone transfer to a held-out domain, so the explanations are not overfit to the eight training domains.","Automatically discovered concepts recover known manual findings (humans weigh clarity, authority, and confidence; LLMs weigh factuality and helpfulness) and add domain-specific concepts such as concealment in safety."],"supporting_citations":[{"why":"Supplies the manually curated 29-concept analysis of human vs LLM preferences that the paper's automatically discovered concepts are compared against.","marker":"Li et al. (2024b)"},{"why":"The dirty model whose shared-plus-task-specific weight decomposition inspires the HMDR objective.","marker":"Jalali et al. (2010)"},{"why":"Provides the Alternative Annotator Test used to statistically validate LLM concept annotations against human annotators.","marker":"Calderon et al. (2025)"},{"why":"Establishes the LLM-as-a-Judge paradigm and the position-bias behavior that Tie Break targets.","marker":"Zheng et al. (2023)"},{"why":"Source of five Reddit-based preference domains and the upvote-based chosen/rejected labeling criteria.","marker":"Ethayarajh et al. (2022)"},{"why":"Provides the PKU safety-focused RLHF preference domain.","marker":"Dai et al. (2024)"},{"why":"Provides the UltraFeedback RLAIF domain used in the eight-domain dataset.","marker":"Cui et al. (2023)"},{"why":"Concept bottleneck models motivate using interpretable concepts as intermediate variables in the pipeline.","marker":"Koh et al. (2020)"}],"fun_headline_variants":["Automated concept discovery explains and predicts preference judgments","White-box model reveals concepts driving human and LLM preferences","Concept explanations that steer LLM outputs and resolve judge ties","From triplets to concepts: interpretable preference prediction across domains"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The concept vocabulary is discovered from human-preference triplets and then assumed to cover the concepts that LLM judges and reward models actually use, with no test of that coverage.","fun_headline_variants_meta":{"raw":{"variants":["Automated concept discovery explains and predicts preference judgments","White-box model reveals concepts driving human and LLM preferences","Concept explanations that steer LLM outputs and resolve judge ties","From triplets to concepts: interpretable preference prediction across domains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000255,"raw_usage":{"total_tokens":1570,"prompt_tokens":942,"completion_tokens":628,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":562}},"tokens_in":558,"tokens_out":628,"duration_ms":7407,"temperature":1.0,"reasoning_tokens":562,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:58:51.301376+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same HMDR pipeline with concepts discovered separately from each explained mechanism's own preference labels and compare accuracy and top concepts against the human-derived 624-concept dictionary; if the mechanism-specific dictionary materially improves prediction or surfaces top concepts absent from the shared set, then the paper's claim that the explanations capture that mechanism's drivers is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The dirty model whose shared-plus-task-specific weight decomposition inspires the HMDR objective."},{"cited_title":"Xing, Hao Zhang, Joseph E","cited_arxiv_id":null,"evidence_quote":"Establishes the LLM-as-a-Judge paradigm and the position-bias behavior that Tie Break targets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of five Reddit-based preference domains and the upvote-based chosen/rejected labeling criteria."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Concept bottleneck models motivate using interpretable concepts as intermediate variables in the pipeline."}],"review_version":1}