{"id":"13a5866a-113d-4fd1-b3c6-588af5b34599","arxiv_id":"2511.19636","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A shared frozen backbone plus per-model LoRA adapters and a concept-diversity loss trains a set of accurate CBMs that reason through different concepts.","lead":"This paper introduces Rashomon Concept Bottleneck Models, a way to train several equally accurate image classifiers that each rely on different human-understandable concepts by adding small adapters to a frozen backbone. It reports that the method produces more diverse explanation sets than baselines while using a fraction of the memory and parameters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Concept annotation quality is the load-bearing risk: if CLIP-generated CIFAR-10 concepts are noisy or incomplete, the reported diversity is diversity over artifacts rather than human-meaningful reasoning.","rationale":"The reader's weakest_assumption identifies the concept annotation quality as the critical unsupported premise for the interpretability claim. My analysis agrees this is the most load-bearing concern: the paper's diversity metrics (concept CKA, SHAP similarity) are defined on the predicted concept vectors and therefore inherit any errors in the concept labels. The method explicitly optimizes diversity of concept predictions, so if the ground-truth concepts are wrong, the optimization will find diverse ways to be wrong. Since CIFAR-10 is a primary demonstration (Table 1) and uses CLIP-generated labels without manual verification, this is a genuine soft spot. The proposed concrete test would settle the concern by comparing against semantics-free and expert-curated concept conditions. The concern is addressable, so the CONDITIONAL verdict remains appropriate; I do not recommend a more severe verdict because the paper also shows results on human-annotated datasets (AwA2, CUB) where this particular issue is mitigated, and the method's efficiency claims appear sound. I agree with the reader's assessment and see no need to change the verdict.","tokens_in":20133,"tokens_out":11262,"duration_ms":124581,"concrete_test":"Repeat the CIFAR-10 Rashomon CBM training under the identical protocol but with two altered concept-label conditions: (a) a random permutation of the CLIP concept labels (destroying semantic correspondence while preserving label statistics), and (b) a human-curated binary concept set for the same images. If task accuracy and diversity metrics in (a) remain comparable to the original results, the diversity does not depend on concept semantics and is over artifacts. If the metrics in (b) differ substantially from the original CLIP-based results, the original diversity measures are sensitive to the specific CLIP annotations. This test isolates whether the reported diversity is meaningful or an artifact of concept-label noise/incompleteness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Rashomon CBMs produce models that 'reason through distinct human-understandable concepts' requires that the concept annotations used in training are correct and complete. This is least secure on CIFAR-10, where concepts are auto-generated by a pretrained CLIP model (Sec. 4.1) with no validation. The diversity regularizer in Eq. (1) explicitly encourages different predicted concept vectors across models. If the concept labels are noisy or miss the features actually driving classification, the models may simply learn to disagree on how to label ambiguous or irrelevant concepts, and the reported concept CKA and SHAP diversity would then reflect disagreement over annotation artifacts rather than semantically meaningful alternative reasoning. The paper provides no evidence that the CLIP concepts for CIFAR-10 are accurate or sufficient; the qualitative analysis (Sec. 4.6) is a single cherry-picked example. While the method also performs well on human-annotated datasets (CUB, AwA2), a general claim about human-understandable diversity cannot rest on an unverified concept set for one of the four datasets.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes 'Rashomon CBMs,' a framework for constructing a set of M concept bottleneck models that all achieve high task accuracy while using different concept-based reasoning. The method freezes one pretrained backbone, inserts model-specific adapters (LoRA for ViT, Conv-Adapter for ResNet), gives each model its own concept heads and label classifier, and jointly trains them with a max-loss objective plus a diversity regularizer on predicted concept vectors. A model-axis gradient checkpointing scheme is introduced to keep memory close to a single-model baseline. Experiments on CIFAR-10, AwA2, CUB, and CelebA with ViT and ResNet backbones compare against random initialization, dropout CBMs, c2y, and x2c baselines, reporting concept/task accuracy, prediction hamming, concept CKA, SHAP similarity, union size, and memory/parameter costs. The paper also includes layer-wise analyses and qualitative examples.","tokens_in":20481,"tokens_out":5107,"duration_ms":58007,"significance":"If the central claims held, this would be a practically useful contribution: it would give a parameter- and memory-efficient way to obtain a diverse Rashomon slice for deep concept-based models, and the model-axis checkpointing idea is interesting. The paper has clear strengths: it evaluates on four datasets and two backbones, uses several diversity metrics, includes layer-wise ablations, and its appendices disclose some limitations (e.g., the CelebA six-concept union collapse). However, the headline empirical claim that the method achieves 'better diversity than baselines' is not uniformly supported by the paper's own tables, and part of the reported diversity is by construction since the diversity loss directly optimizes cosine similarity between predicted concept vectors. The human-understandability claim is also load-bearing and rests on concept annotations that are unvalidated for CIFAR-10 and on selectively chosen qualitative examples.","major_comments":[{"comment":"The abstract's 'better diversity than baselines' and the table captions' 'consistently better' / 'on par with or better than x2c' are contradicted by the reported numbers. In Table 1 (AwA2 ViT), SHAP similarity for x2c is 0.1560 vs. 0.2046 for Rashomon CBMs, and lower is better. In Table 3 (AwA2 ResNet), concept CKA for x2c is 0.349 vs. 0.674 for Rashomon CBMs, again lower is better. Table 4 (CUB ViT) shows concept CKA 0.4698 for x2c vs. 0.6416 for Rashomon CBMs. These are not isolated small gaps; they are the primary diversity metrics in three different configurations. The central empirical claim must be qualified to 'sometimes better, sometimes worse,' and the caption wording corrected.","section":"Abstract; §4.2 Table 1; §4.3 Table 3"},{"comment":"The diversity loss in Eq. (1) directly minimizes the average pairwise cosine similarity between predicted concept vectors. The paper then reports 'concept similarity' and concept CKA trends as evidence of diversity (e.g., Fig. 2 and Fig. 8). These quantities are essentially the training objective, so observing that they change when the diversity loss is used is by construction, not a discovery about model behavior. The more meaningful evidence for behavioral diversity is SHAP similarity and union size, but those results are mixed, as noted above. To support the central claim that the models 'reason through distinct concepts,' the paper should present SHAP-based and top-concept-union diversity as the primary outcome, and should include an ablation that isolates the effect of the architecture (adapters) from the effect of L_div, e.g., training with L_div removed or with L_div but no adapte","section":"Eq. (1); §4.2; Figs. 2 and 8"},{"comment":"The claim that Rashomon CBMs produce models that reason through 'distinct human-understandable concepts' depends on the correctness and completeness of the concept annotations. For CIFAR-10, concepts are auto-generated by a pretrained CLIP model (§4.1) with no validation or error analysis. If those labels are noisy or miss the features driving classification, the measured diversity is diversity over annotation artifacts rather than over human-meaningful reasoning. The qualitative support is also weak: §4.6 shows one manually chosen example, and Appendix D explicitly states that five of ten models were selected 'because they exhibit the most informative and diverse patterns.' The paper should provide a systematic validation of the CIFAR-10 concepts (e.g., human evaluation or comparison with human-annotated concept sets) and should present non-selectively chosen qualitative examples, or ex","section":"§4.1; §4.6; Appendix D"},{"comment":"The abstract and conclusion claim that the framework 'demonstrates' trustworthy model selection, resolution of inter-class confusion, and reliable abstention. The experimental section, however, contains no experiments for abstention, no inter-class-confusion experiments, and no task other than classification with diversity metrics. These applications are only mentioned as future motivations. Either the paper should add experiments supporting these claims, or these sentences should be removed from the abstract and conclusion, since they are not demonstrated by the reported results.","section":"Abstract; §5; §4"}],"minor_comments":[{"comment":"The symbol α is used both for the diversity weight in Eq. (1) and for the LoRA scaling factor (α=16) in Section 3.2.3 and the implementation details. These are different quantities; rename one to avoid confusion.","section":"§3.2.2 vs §3.2.3"},{"comment":"Several table captions say the method 'consistently achieves better' diversity or performs 'on par with or better than x2c.' As noted, this is not true for several rows. The captions should be revised to accurately describe the direction and variability of the comparisons.","section":"Tables 1, 4, 6"},{"comment":"Typo: 'diferent' should be 'different.' Also in Appendix C, 'there is might be no diversity' should be 'there might be no diversity.'","section":"Appendix A.2"},{"comment":"The qualitative example manually picks 15 concepts and later says 'we show 5 models out of 10.' The selection criteria for both the image and the concepts should be stated in the main text, not deferred to Appendix D, where the models are described as selected for being 'most informative and diverse.'","section":"§4.6"},{"comment":"The description of the dynamic α schedule is brief. It would help to state whether α is annealed by the equation as written, and how the schedule interacts with the reported results across runs with different initial α values.","section":"§3.2.2 and Eq. (1)"}],"recommendation":"major_revision","confidential_remarks":"This is a useful engineering contribution, and the multi-dataset, two-backbone evaluation is a strength. However, the abstract and several captions overstate the empirical support: the method is not uniformly better than x2c on the primary diversity metrics, and some diversity is by construction. The human-meaningfulness claim rests on unvalidated CLIP concepts for CIFAR-10 and on selectively chosen examples. I would not reject: the issues are fixable with qualified claims, corrected captions, additional validation of the concept annotations, and either new application experiments or removal of the corresponding abstract claims. The novelty relative to LoRA-ensemble work and Zhang et al. [50] is moderate, but the CBM-specific Rashomon-slice construction and the model-axis checkpointing are worth publishing if the claims are made precise."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The genuinely new piece is the engineering combination: frozen shared backbone, per-model LoRA adapters, a max-loss over models plus a diversity regularizer on predicted concept vectors, and model-axis checkpointing. That last piece is the sleeper hit—memory at 7.6% of full retraining is a real practical gain, and it means you can actually explore M=25 models on one GPU. The paper does that job well, and the experiments span four datasets and two backbones. Credit where due: the layer-wise ablation showing diversity concentrates in deep adapters is a nice, informative result, not just a checkmark.\n\nThe soft spots are real but not fatal. The abstract says 'better diversity than baselines' and that's not uniform: x2c beats them on SHAP similarity on AwA2 ViT and on concept CKA on ResNet AwA2. That's a claim-to-metric mismatch, fixable in revision. More substantially, the diversity loss directly minimizes cosine similarity between predicted concept vectors, so the concept-similarity and CKA numbers are partly measuring whether the optimizer did its job, not whether the resulting models are genuinely different in their reasoning. The SHAP and classifier-weight analyses are the evidence that matters, and they're suggestive but limited—one cherry-picked tiger image plus a couple of appendix examples.\n\nThe stress-test concern about CIFAR-10 concept annotations is legitimate. The paper uses CLIP-generated concepts without any validation, and if those labels are noisy or incomplete, the measured diversity there is diversity over artifacts. The authors should at least acknowledge this, ideally validate a sample or rely more on CUB/AwA2 where concepts are human-annotated. That said, the central construction—diverse accurate CBMs via adapters—does not collapse without CIFAR-10; it just weakens the 'human-understandable' framing for that dataset.\n\nMissing code and data are a genuine issue for a methods paper. Also, the abstract promises abstention and inter-class confusion benefits that are never tested; those claims should be removed or substantiated.\n\nBottom line: this is a solid, useful contribution to the Rashomon-set-for-deep-models literature. It deserves a serious referee. I'd want the revision to tone down the abstract, add code, validate or caveat the CIFAR-10 concepts, and either test or drop the downstream-benefit claims.","headline":"Useful, memory-efficient method for building diverse concept bottleneck models; empirical claims mostly hold but abstract overstates and CIFAR-10 concept quality is a real risk.","tokens_in":20913,"tokens_out":2022,"would_cite":true,"duration_ms":19595,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T05","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a family of equally accurate concept-based models that reason through visibly different concepts can be trained in a single run, at nearly the memory cost of one model.","keywords":["Rashomon set","concept bottleneck models","model multiplicity","interpretability","adapters","low-rank adaptation","gradient checkpointing","concept diversity"],"falsifier":"Zero out a member model's top SHAP concept at test time — e.g., remove 'meat' from a tiger-classifying model whose decisions are SHAP-dominated by 'meat' — and measure how much predictions and accuracy move; if they barely move, the model does not actually rely on that concept and the claimed diverse reasoning is not behaviorally real. Applying the same intervention to the auto-generated CIFAR-10 concepts would test whether the diversity survives when label quality is unverified.","tokens_in":20049,"feed_emoji":"🔀","tokens_out":11763,"duration_ms":105173,"temperature":0.7,"pith_summary":"The paper sets out to make the Rashomon set — the family of near-equally accurate models for a task — visible and usable for deep, interpretable models instead of leaving it a theoretical abstraction. Its central claim is that a diverse Rashomon slice of Concept Bottleneck Models (CBMs, which predict through human-understandable intermediate concepts) can be constructed in one training run: M models share a single frozen backbone, each carries its own lightweight adapters and its own concept-to-label head, and a min-max loss with a concept-diversity regularizer keeps every member accurate while pushing their concept representations apart. On four image datasets the resulting models match or beat baseline accuracy while agreeing far less on which concepts drive their predictions, and a model-axis checkpointing scheme keeps memory near that of a single model. If correct, this turns 'how many ways can a model be right?' into a practical audit tool: choose among equally valid reasoning styles, detect models leaning on spurious cues, and abstain when the slice disagrees.","feed_headline":"One training run builds many accurate models that reason differently","feed_subtitle":"Equally accurate models that split on which concepts drive predictions — a new handle for AI auditing.","key_machinery":"The central construction is the parallel-adapter slice: one frozen backbone plus M model-specific adapter sets, each feeding its own concept heads and linear classifier, trained jointly under L_total = max_m L_pr + λ (max_m L_c − α/M Σ_m L_div), where the diversity term penalizes cosine similarity between members' predicted concept vectors. The max-operations keep every member accurate; the diversity term prevents collapse to one reasoning path; α is scheduled from gradient magnitudes. Model-axis gradient checkpointing wraps each model's full forward–backward pass in a checkpoint boundary so one model holds activations at a time, keeping memory near that of a single model rather than linear","core_discovery":"The core claim: a Rashomon slice of CBMs — equally accurate models that reason through distinct human-understandable concepts — can be produced in one joint training run. Each of M models has its own low-rank adapters on a shared frozen backbone, concept predictors, and classifier. Joint training minimizes the worst member's task and concept losses while subtracting a diversity term that penalizes cosine similarity between predicted concept vectors, keeping every member accurate yet spread across concept space. On four image datasets the slice matches or beats baseline accuracy while showing the lowest concept-representation and concept-usage similarity; the paper applies the slice to model","pith_inferences":["Editorial inference: diversity is measured on predicted concept vectors, so semantically overlapping concepts (e.g., 'jungle' vs. 'bush') could inflate the reported diversity; a distance metric over concept meaning would reveal how much of it is substantive.","Editorial inference: the parallel-adapter recipe could transfer beyond CBMs to any architecture with an inspectable intermediate layer — for instance, hidden states of a language model — though the 'human-understandable' guarantee would then rest on that layer's interpretability.","Editorial inference: the layer-wise finding that deep adapters carry most of the diversity suggests a cheaper configuration the paper does not recommend — share early adapters across members and differentiate only the final blocks — which the ablation indirectly supports."],"forward_implications":["A single joint training run yields a full slice of accurate CBMs with visibly different concept reliance, so practitioners can compare and select among equally valid reasoning strategies rather than retraining many models.","Diversity concentrates in the adapters of the last few blocks while low-level features stay shared, giving a layer-wise account of how accuracy and diversity coexist.","The slice supports practical trust tools: reliable abstention when members disagree, and identification of models that lean on contextual or potentially spurious concepts such as 'jungle' or 'meat' for a tiger.","Memory stays near that of a single model, so the slice can grow (tested from 10 to 25 members) without exceeding GPU limits, and task accuracy remains stable as it grows.","The construction is backbone-agnostic, reproducing accurate, diverse slices with both ViT/LoRA and ResNet-18/Conv-Adapter instantiations."],"fun_headline_variants":["Diverse but equally accurate models from one training run","Rashomon set for concept models: diverse logic, equal accuracy","One joint run, many concept models: each accurate, each different","Train once, get many equally accurate concept models","Explore the Rashomon set: one run yields many accurate models"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The claim that the models 'reason through distinct human-understandable concepts' rests on the concept labels being correct and complete, especially the CLIP-auto-generated CIFAR-10 concepts (Sec. 4.1), because all diversity metrics live in the space of those predicted concepts; noisy or superficial labels would make the measured diversity a diversity of artifacts, not of human-meaningful reasoning.","fun_headline_variants_meta":{"raw":{"variants":["Diverse but equally accurate models from one training run","Rashomon set for concept models: diverse logic, equal accuracy","One joint run, many concept models: each accurate, each different","Train once, get many equally accurate concept models","Explore the Rashomon set: one run yields many accurate models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001073,"raw_usage":{"total_tokens":4302,"prompt_tokens":686,"completion_tokens":3616,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":430,"completion_tokens_details":{"reasoning_tokens":3533}},"tokens_in":430,"tokens_out":3616,"duration_ms":24202,"temperature":1.0,"reasoning_tokens":3533,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T20:27:11.690681+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Zero out a member model's top SHAP concept at test time — e.g., remove 'meat' from a tiger-classifying model whose decisions are SHAP-dominated by 'meat' — and measure how much predictions and accuracy move; if they barely move, the model does not actually rely on that concept and the claimed diverse reasoning is not behaviorally real. Applying the same intervention to the auto-generated CIFAR-10 concepts would test whether the diversity survives when label quality is unverified.","supporting_citations":[],"review_version":1}