Pith. sign in

REVIEW 5 major objections 5 minor 40 references

SEG-MIL-CBM claims that decomposing images into concept-guided segments and aggregating them with attention improves worst-group accuracy over global-concept CBMs while yielding spatially grounded explanations.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-04 11:29 UTC pith:SVCGCNMG

load-bearing objection Good idea, unreliable numbers: the abstract and Table 3 disagree on the main Waterbirds result, and the error bars look too small to be real. the 5 major comments →

arxiv 2510.04180 v2 pith:SVCGCNMG submitted 2025-10-05 cs.CV cs.LG

Spatially Grounded Concept-Based Image Classification

Ran Eisenberg , Amit Rozner , Ethan Fetaya , Ofir Lindenbaum This is my paper
classification cs.CV cs.LG
keywords concept bottleneck modelsmultiple instance learningspurious correlationworst-group accuracyspatial groundingattention-based aggregationfoundation modelsinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Concept Bottleneck Models expose human-interpretable concepts but reason globally, so they can still lean on spurious background cues. This paper proposes SEG-MIL-CBM, which first segments each image into concept-guided regions via a CLIP–GroundingDINO–SAM pipeline, then treats each region as a bag instance in an attention-based multiple instance learning framework. The attention weights both produce the prediction and identify which regions support it, so the explanation and the decision share the same evidence. The paper reports that this design raises worst-group accuracy on Waterbirds and Pawrious relative to CBM baselines, without any group annotations, while staying competitive on ImageNet, Places, CUB, and CIFAR-100.

Core claim

On the paper's own terms, the central discovery is that spatial grounding alone—forcing the classifier to aggregate evidence from concept-aligned regions rather than global image features—can suppress spurious cues that standard CBMs absorb. The model's prediction is a weighted sum of segment-level concept activations, with weights learned by attention; those same weights and activations constitute the explanation. The authors report that this yields the best CBM accuracy on CIFAR-100 (85.3%) and improves worst-group accuracy on Waterbirds and Pawrious compared with prior CBMs, approaching or exceeding group-supervised baselines without ever seeing group labels.

What carries the argument

The key machinery is concept-guided segmentation plus attention-based multiple instance learning. A pretrained CLIP scores each image against a concept vocabulary; the top-K concepts are fed to GroundingDINO to propose bounding boxes, and SAM converts each box into a binary mask. Each segment is an instance carrying a CLIP concept similarity vector. A linear concept head projects segment features into concept space, an attention module learns a weight per segment, and the weighted sum of concept activations feeds a linear classifier. A cosine-similarity loss aligns predicted segment concepts with the CLIP similarity vectors, letting the model highlight task-relevant segments and down-weight

Load-bearing premise

The entire approach depends on the reliability of the concept-guided segmentation pipeline: if CLIP, GroundingDINO, or SAM fail to localize the task-relevant object (due to occlusion, tiny objects, or a concept vocabulary that lacks the relevant class), the bag of instances omits the evidence needed for correct classification.

What would settle it

Retrain SEG-MIL-CBM on Waterbirds with the concept-guided segmentation replaced by random boxes of the same size distribution; if worst-group accuracy stays high, the spatial grounding from foundation models is not what drives the gain. Alternatively, construct a dataset where the class-defining feature is deliberately absent from the concept vocabulary; if the model still classifies accurately, the concept bottleneck is not the actual decision mechanism.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If correct, interpretable concept-bottleneck models can match or exceed the worst-group robustness of methods that use explicit group labels, without any group supervision.
  • Explanations become intrinsic to the decision: the same segment weights and concept activations that produce the logit are the explanation, so the model cannot hide reliance on spurious regions behind a separate attribution module.
  • Fine-grained recognition (CUB, CIFAR-100) may benefit from segment-level reasoning because localized parts like beaks, crowns, or wing patterns are naturally isolated by the segmentation pipeline.
  • Robustness to common corruptions improves at high severities, consistent with attention down-weighting background regions that degrade under noise and blur.
  • The framework is portable to any dataset with a concept vocabulary and an open-vocabulary detector, offering a path to interpretable, robust models at scale.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable implication is that the segmentation quality, not the MIL aggregation itself, drives the worst-group gain; ablating to random boxes or ground-truth boxes would disentangle these factors.
  • Because the bag construction is frozen and relies on CLIP's top-K concept selection, the approach inherits CLIP's concept-vocabulary biases; classes whose discriminative features fall outside the vocabulary would be invisible to the model, a limitation the paper does not directly address.
  • The paper's abstract and Table 3 report different magnitudes for the Waterbirds worst-group improvement (65.1→72.0 versus 54.6→85.5); if the larger table numbers are the correct reading, the robustness gain is much stronger than the abstract's headline suggests, and the abstract's 'from 65.1 to 72.0' may refer to a different comparison set.
  • A natural extension is to jointly learn the concept vocabulary with the segmentation module, replacing the fixed CLIP-based concept list with a learned one to reduce dependence on the initial vocabulary.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes SEG-MIL-CBM, a concept bottleneck model that decomposes each image into concept-guided regions via CLIP, GroundingDINO, and SAM, and then treats those regions as instances in an attention-based multiple-instance-learning (MIL) framework. Segment-level concept activations are aligned with CLIP similarity scores through a cosine concept-alignment loss, and the attention-weighted aggregation is used both for classification and for producing spatially grounded, concept-level explanations. The authors claim improved worst-group accuracy over CBM-family baselines on Waterbirds and on a newly introduced 'Pawrious' benchmark, competitive standard accuracy on ImageNet/Places/CUB/CIFAR, corruption robustness on CIFAR-10-C, and faithful segment-level explanations on CUB, all without group or concept annotations.

Significance. If the empirical claims hold, the paper offers a useful synthesis of concept-bottleneck interpretability and attention-based multiple-instance learning: the model explicitly ties localized visual evidence to concept-level reasoning and does not require group labels. The idea of using foundation-model segmentation to build bags of concept instances is timely, and the authors evaluate on multiple benchmarks, which is a strength. However, the central robustness claim is currently undermined by (i) direct numerical contradictions between the abstract and the main tables, (ii) implausibly small reported standard deviations, (iii) the absence of the CUB segment-level faithfulness experiments promised in the abstract, and (iv) an incompletely specified new benchmark. These issues are load-bearing for the paper's headline results and must be resolved before the contribution can be assessed.

major comments (5)
  1. [Abstract; Tables 3 and 9] The abstract's headline numbers do not match the body. The abstract states Waterbirds worst-group accuracy improves 'from 65.1% to 72.0%,' but Table 3 reports SEG-MIL-CBM at 85.54±0.005, with Label-Free-CBM at 54.62 and Post-hoc-CBM at 57.89. No 72.0 value appears anywhere in the paper; the closest match is Label-Free-CBM's CIFAR-100 accuracy of 65.1 in Table 9, which is a different quantity. A reader cannot determine which result is authoritative. Please reconcile the abstract with the tables and state the exact experimental setting for the headline claim.
  2. [§5.4 and Tables 2, 3, 7, 9] The reported standard deviations are not plausible for three independent training runs. For example, Table 3 reports Waterbirds worst-group accuracy as 85.54±0.005 percentage points; with ~5,794 Waterbirds test images, the binomial standard error would be on the order of 0.5 percentage points, about 100 times larger. CIFAR-100 in Table 2 is reported as 85.26±0.00. Moreover, §5.4 says 'we report means only in all tables' and refers the reader to 'full per-seed statistics' in Appendix C, but Appendix C contains mean accuracies rather than per-seed values. Please report the actual per-seed numbers, or state explicitly if the numbers come from a deterministic/degenerate procedure.
  3. [Abstract; §6; Appendix C] The abstract promises 'Segment-level faithfulness experiments on CUB further show that its learned segment ranking matches or improves over evaluated segment-ranking controls,' but the full text contains no such experiment. Section 6 reports recognition accuracy, CIFAR-10-C robustness, and a comparison to group-robust baselines, but no segment-ranking metric, no faithfulness baseline, and no CUB faithfulness study. This evidence is central to the interpretability claim in the title and framing. The experiment should be added, or the claim should be removed and the interpretability evidence explicitly limited to what is actually shown.
  4. [§5.1; Tables 3 and 4] The paper introduces 'Pawrious' as a new dataset but only describes it as 'a customized variant of the Stable Diffusion Spawrious framework' and cites Spawrious [17]. No dataset size, class composition, group structure, generation procedure, or release URL is given. Since Pawrious is the second empirical pillar of the robustness claim, the benchmark is not reproducible as written. Please provide full specification and, if the dataset is publicly available, a link; if it is not, this should be stated clearly as a limitation.
  5. [§4, concept alignment loss; §2] The concept-alignment loss trains the concept head to match CLIP similarity scores, and the same CLIP model selects the top-K concepts and, through GroundingDINO/SAM, determines the spatial segments. The resulting explanations are, at least in part, a distillation of CLIP's priors rather than an independent semantic grounding. This does not invalidate the approach, but the paper's language — e.g., 'causally relevant concepts' and 'spatially grounded, concept-level explanations' — overstates what is demonstrated. At minimum, this dependency should be acknowledged as a limitation, and the interpretability claim should be supported by an independent evaluation (e.g., human-annotation agreement or concept-intervention tests), preferably the CUB faithfulness study promised in the abstract.
minor comments (5)
  1. [Title] The arXiv title 'Spatially Grounded Concept-Based Image Classification' differs from the full-text title 'From Segments to Concepts: Interpretable Image Classification via Concept-Guided Segmentation.' Please align these titles.
  2. [§5.3 and Appendix A] Several hyperparameters are not specified: K_top, tau_minpix, tau_IoU, number of epochs, batch size, and the 'easy/hard batch alternation' details. Please add these values for reproducibility.
  3. [§5.4 vs Tables 3/7] The text says 'we report means only in all tables,' but Tables 3 and 7 include ± standard deviations. Please reconcile this statement.
  4. [Appendix C, Table 9] The table caption 'Full Per-Seed Statistics' is misleading because the table reports means, not per-seed values. Please provide actual per-seed numbers or rename the table.
  5. [§5.3; Figure 4] The confidence intervals in Figure 4 are described as 95% over five corruptions, but the number of runs/seeds used to compute them is not stated. Specify this, and consider including all corruptions in the main figure rather than only five.

Circularity Check

0 steps flagged

No significant circularity: the robustness and accuracy claims are learned from external image-level labels, and the CLIP-based concept alignment is an acknowledged auxiliary objective rather than a fitted input renamed as a prediction.

full rationale

The paper's derivation chain is not circular. SEG-MIL-CBM trains its attention module and classifier on image-level class labels via the classification loss L_cls, so the main predictive and worst-group accuracy claims are grounded in external supervision, not in the model's own outputs. The concept alignment loss L_concept ('aligns predicted segment-level concept activations z~_i with the corresponding CLIP similarity vectors z^CLIP_i') is an explicitly stated auxiliary regularizer; it encourages semantic consistency but does not define the classification prediction. The segmentation pipeline (CLIP top-K selection, GroundingDINO detection, SAM masks) is a fixed preprocessing step that constructs the bag of instances; it is a representation choice, not a fitted parameter later reported as a prediction. No load-bearing self-citation or imported uniqueness theorem is used: the authors' own prior works appear only in future-work and related-work contexts. The paper does contain separate verification concerns—the abstract's Waterbirds improvement (65.1% to 72.0%) does not match Table 3's numbers (54.62% and 57.89% to 85.54%), and the abstract-promised CUB faithfulness experiments are absent from the provided full text—but these are consistency/evidence issues, not circularity. The explanations are internal model outputs; that they are partially aligned with CLIP is a design property, not a circular derivation of the central empirical claims.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 1 invented entities

The central claim depends on a handful of tuning hyperparameters and several strong assumptions about the reliability of the foundation-model segmentation pipeline and the semantic validity of CLIP scores. The method introduces one new dataset that is not publicly released.

free parameters (5)
  • λ_concept = 0.1
    Weight of the concept alignment loss; no ablation is provided, and it directly controls the trade-off between classification and CLIP-alignment of concept activations.
  • Bag size N_s = 15 (Waterbirds/Pawrious), 5 (CIFAR10/Places/ImageNet)
    Number of segments per image; affects attention pooling and computation; set per dataset without analysis.
  • ρ_max = 0.5
    Maximum mask area ratio; segments covering more than 50% of the image are discarded, potentially removing large relevant objects.
  • K_top = unspecified
    Number of top concepts selected per image by CLIP similarity; not reported despite being a core step in Algorithm 1.
  • τ_minpix, τ_IoU = unspecified
    Minimum mask pixel threshold and IoU merging threshold; both affect bag composition and are never quantified.
axioms (4)
  • domain assumption CLIP, GroundingDINO, and SAM provide reliable concept-guided segmentation across all evaluated datasets.
    The entire bag construction depends on these frozen foundation models; if they fail to segment the object, the bag lacks task evidence. Acknowledged in Limitations.
  • domain assumption The Label-Free-CBM concept vocabulary is sufficient for all evaluated datasets.
    Concepts are drawn from Label-Free-CBM's released list; if a class is not represented, the model may not capture it.
  • domain assumption CLIP similarity scores provide a valid semantic target for concept activations.
    The concept alignment loss uses CLIP scores as ground truth, assuming they reflect human-interpretable semantics; this may propagate CLIP biases into explanations.
  • domain assumption A single attention vector w can learn to suppress spurious segments.
    The attention mechanism is a linear scoring function over segment features; if relevant and spurious segments have similar features, attention cannot separate them.
invented entities (1)
  • Pawrious dataset no independent evidence
    purpose: A new benchmark for spurious correlations in dog breed classification with two classes (companion vs. working dogs).
    Introduced in this paper, derived from Spawrious, but no release details or external validation are provided.

pith-pipeline@v1.3.0-alltime-deepseek · 14953 in / 15046 out tokens · 100317 ms · 2026-08-04T11:29:10.600117+00:00 · methodology

0 comments
read the original abstract

Deep neural networks can achieve high accuracy while relying on evidence that is hard to inspect or misaligned with the intended task. Concept Bottleneck Models (CBMs) expose human-interpretable concepts, but most treat concepts as global attributes and do not show how localized evidence is aggregated into a decision. We propose \textbf{SEG-MIL-CBM}, a spatially grounded CBM that decomposes each image into concept-guided regions and classifies it by attention-based aggregation of segment-level concept evidence. The same segment evidence terms form the prediction and the explanation, exposing which regions and concepts support the predicted logit without a separate post-hoc attribution module. Among evaluated CBM-family baselines, SEG-MIL-CBM improves Waterbirds worst-group accuracy from $65.1\%$ to $72.0\%$, reaches $87.4\%$ worst-group accuracy on Pawrious, remains competitive on standard recognition, and attains the best CBM accuracy on CIFAR-100 ($85.3\%$). Segment-level faithfulness experiments on CUB further show that its learned segment ranking matches or improves over evaluated segment-ranking controls.

Figures

Figures reproduced from arXiv: 2510.04180 by Amit Rozner, Ethan Fetaya, Ofir Lindenbaum, Ran Eisenberg.

Figure 1
Figure 1. Figure 1: Overview of Concept Bottleneck Mod￾els (CBM) versus our proposed SEG-MIL-CBM. (a) CBMs predict labels using concept bottleneck lay￾ers, which are derived at the global image level. (b) SEG-MIL-CBM first segments the image into seman￾tically meaningful regions and treats each as an in￾stance in an attention-based multiple instance learning framework. This enables the model to identify task￾relevant regions,… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of our concept-guided segmentation pipeline. Given an input image, CLIP [24] Image Encoder extracts image embeddings while a concept set is encoded by CLIP [24] Text Encoder. The top-Ktop concepts most relevant to the image are selected by cosine-similarity scores. They are then used with GroundingDINO [16] and SAM [11] to produce semantically meaningful segments. Each segment is annotated with co… view at source ↗
Figure 3
Figure 3. Figure 3: Overview of the SEG-MIL-CBM training pipeline. Each input image is decomposed into concept-guided segments {s1, . . . , sNs }, which are passed through a shared backbone to produce features hi = ϕ(si). These features are projected into a concept space via Z = WcH, and segment-level activations are aligned with CLIP-derived similarity vectors using a similarity-based concept loss. An attention mechanism ass… view at source ↗
Figure 4
Figure 4. Figure 4: 95% confidence interval accuracy across 5 [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Accuracy trends across corruption types (page 1) [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Accuracy trends across corruption types (page 2) [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 7 linked inside Pith

  1. [1]

    Show and tell: Visually explainable deep neural nets via spatially- aware concept bottleneck models.arXiv preprint arXiv:2502.20134, 2025

    Itay Benou and Tammy Riklin-Raviv. Show and tell: Visually explainable deep neural nets via spatially- aware concept bottleneck models.arXiv preprint arXiv:2502.20134, 2025. 2, 3, 4, 7, 8, 16

    arXiv 2025

  2. [2]

    Mechanistic interpretability for ai safety–a review.arXiv preprint arXiv:2404.14082, 2024

    Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety–a review.arXiv preprint arXiv:2404.14082, 2024. 2

    Pith/arXiv arXiv 2024

  3. [3]

    Emerging properties in self-supervised vi- sion transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv´ e J´ egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vi- sion transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021. 3

    2021

  4. [4]

    Adaptive concept bottle- neck for foundation models

    Jihye Choi, Jayaram Raghuram, Yixuan Li, Suman Banerjee, and Somesh Jha. Adaptive concept bottle- neck for foundation models. InICML 2024 Workshop on Foundation Models in the Wild, 2024. 3

    2024

  5. [5]

    Environment inference for invariant learning

    Elliot Creager, J¨ orn-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. InInternational Conference on Machine Learning, pages 2189–2200. PMLR, 2021. 3, 4, 6, 8, 16

    2021

  6. [6]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on com- puter vision and pattern recognition, pages 248–255. Ieee, 2009. 6

    2009

  7. [7]

    Coper: Correlation-based permutations for multi-view clustering

    Ran Eisenberg, Jonathan Svirsky, and Ofir Linden- baum. Coper: Correlation-based permutations for multi-view clustering. InThe Thirteenth Interna- tional Conference on Learning Representations. 9

  8. [8]

    Benchmark- ing neural network robustness to common corruptions and perturbations.Proceedings of the International Conference on Learning Representations, 2019

    Dan Hendrycks and Thomas Dietterich. Benchmark- ing neural network robustness to common corruptions and perturbations.Proceedings of the International Conference on Learning Representations, 2019. 6, 8

    2019

  9. [9]

    Attention-based deep multiple instance learning

    Maximilian Ilse, Jakub M Tomczak, and Max Welling. Attention-based deep multiple instance learning. InInternational Conference on Machine Learning (ICML), pages 2132–2141, 2018. 3

    2018

  10. [10]

    Last layer re-training is sufficient for robustness to spurious correlations.arXiv preprint arXiv:2204.02937, 2022

    Polina Kirichenko, Pavel Izmailov, and Andrew Gor- don Wilson. Last layer re-training is sufficient for robustness to spurious correlations.arXiv preprint arXiv:2204.02937, 2022. 3, 4, 6, 8, 16

    Pith/arXiv arXiv 2022

  11. [11]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. InProceedings of the IEEE/CVF international conference on computer vi- sion, pages 4015–4026, 2023. 3, 4, 5

    2023

  12. [12]

    Concept bottleneck models

    Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. InIn- ternational conference on machine learning, pages 5338–5348. PMLR, 2020. 2, 3

    2020

  13. [13]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009. 6

    2009

  14. [14]

    Learning coupled embedding using multiview diffu- sion maps

    Ofir Lindenbaum, Arie Yeredor, and Moshe Salhov. Learning coupled embedding using multiview diffu- sion maps. InInternational Conference on Latent Variable Analysis and Signal Separation, pages 127–

  15. [15]

    Just train twice: Improving group robustness without training group information

    Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. InInternational Conference on Machine Learning, pages 6781–6792. PMLR, 2021. 3, 4, 6, 7, 8, 16

    2021

  16. [16]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jian- wei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. InEuropean Conference on Computer Vi- sion, pages 38–55. Springer, 2024. 3, 4, 5

    2024

  17. [17]

    Spawrious: A benchmark for fine control of spurious correlation biases.arXiv preprint arXiv:2303.05470, 2023

    Aengus Lynch, Gb` etondji JS Dovonon, Jean Kaddour, and Ricardo Silva. Spawrious: A benchmark for fine control of spurious correlation biases.arXiv preprint arXiv:2303.05470, 2023. 6

    Pith/arXiv arXiv 2023

  18. [18]

    Visual classification via description from large language models.arXiv preprint arXiv:2210.07183, 2022

    Sachit Menon and Carl Vondrick. Visual classification via description from large language models.arXiv preprint arXiv:2210.07183, 2022. 4, 6, 8, 16

    Pith/arXiv arXiv 2022

  19. [19]

    Decompose-and- compose: A compositional approach to mitigating spurious correlation

    Fahimeh Hosseini Noohdani, Parsa Hosseini, Aryan Yazdan Parast, Hamidreza Yaghoubi Araghi, and Mahdieh Soleymani Baghshah. Decompose-and- compose: A compositional approach to mitigating spurious correlation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 27662–27671, June 2024. 3, 4, 6, 8, 16

    2024

  20. [20]

    Label-free concept bottleneck models

    Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. InInternational Conference on Learning Representations, 2023. 2, 3, 4, 5, 6, 7, 8, 9, 16

    2023

  21. [21]

    Sparse linear concept discovery models

    Konstantinos Panagiotis Panousis, Dino Ienco, and Diego Marcos. Sparse linear concept discovery models. InProceedings of the ieee/cvf international conference on computer vision, pages 2767–2771, 2023. 4, 6, 8, 16

    2023

  22. [22]

    DCBM: Data-efficient visual concept bottleneck models

    Katharina Prasse, Patrick Knab, Sascha Marton, Christian Bartelt, and Margret Keuper. DCBM: Data-efficient visual concept bottleneck models. In International Conference on Machine Learning, 2025. 2, 3, 4, 7, 8, 16

    2025

  23. [23]

    Simple and fast group robustness by automatic feature reweighting

    Shikai Qiu, Andres Potapczynski, Pavel Izmailov, and Andrew Gordon Wilson. Simple and fast group robustness by automatic feature reweighting. InIn- ternational Conference on Machine Learning, pages 28448–28467. PMLR, 2023. 3, 4, 6, 7, 8, 16

    2023

  24. [24]

    Learning transferable visual models from natu- ral language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natu- ral language supervision. InInternational conference on machine learning, pages 8748–8763. PmLR, 2021. 2, 4, 5

    2021

  25. [25]

    Discover-then-name: Task-agnostic concept bottlenecks via automated concept discovery

    Sukrut Rao, Sweta Mahajan, Moritz B¨ ohle, and Bernt Schiele. Discover-then-name: Task-agnostic concept bottlenecks via automated concept discovery. InEuro- pean Conference on Computer Vision, pages 444–461. Springer, 2024. 4, 6, 8, 16 10

    2024

  26. [26]

    Domain-generalizable multiple-domain clustering.Transactions on Machine Learning Re- search

    Amit Rozner, Barak Battash, Lior Wolf, and Ofir Lindenbaum. Domain-generalizable multiple-domain clustering.Transactions on Machine Learning Re- search. 9

  27. [27]

    Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization.arXiv preprint arXiv:1911.08731, 2019

    Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization.arXiv preprint arXiv:1911.08731, 2019. 1, 3, 4, 6, 7, 8, 16

    Pith/arXiv arXiv 1911

  28. [28]

    Multi-view kernel consensus for data analysis.Applied and Computational Harmonic Analysis, 49(1):208– 228, 2020

    Moshe Salhov, Ofir Lindenbaum, Yariv Aizenbud, Avi Silberschatz, Yoel Shkolnisky, and Amir Averbuch. Multi-view kernel consensus for data analysis.Applied and Computational Harmonic Analysis, 49(1):208– 228, 2020. 9

    2020

  29. [29]

    Deep inside convolutional networks: Visualising image classification models and saliency maps.arXiv preprint arXiv:1312.6034, 2013

    Karen Simonyan, Andrea Vedaldi, and Andrew Zisser- man. Deep inside convolutional networks: Visualising image classification models and saliency maps.arXiv preprint arXiv:1312.6034, 2013. 2

    Pith/arXiv arXiv 2013

  30. [30]

    Contextual feature selection with conditional stochastic gates

    Ram Dyuthi Sristi, Ofir Lindenbaum, Shira Lifshitz, Maria Lavzin, Jackie Schiller, Gal Mishne, and Hadas Benisty. Contextual feature selection with conditional stochastic gates. InInternational Conference on Ma- chine Learning, pages 46375–46392. PMLR, 2024. 2

    2024

  31. [31]

    Interpretable deep clustering for tabular data

    Jonathan Svirsky and Ofir Lindenbaum. Interpretable deep clustering for tabular data. InInternational Conference on Machine Learning, pages 47314–47330. PMLR, 2024. 2

    2024

  32. [32]

    The caltech- ucsd birds-200-2011 dataset

    Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech- ucsd birds-200-2011 dataset. Technical Report CNS- TR-2011-001, California Institute of Technology, Pasadena, CA, 2011. 6

    2011

  33. [33]

    Discover and cure: Concept-aware miti- gation of spurious correlation

    Shirley Wu, Mert Yuksekgonul, Linjun Zhang, and James Zou. Discover and cure: Concept-aware miti- gation of spurious correlation. InICML, 2023. 3, 4, 6

    2023

  34. [34]

    Locally sparse neural networks for tabular biomed- ical data

    Junchen Yang, Ofir Lindenbaum, and Yuval Kluger. Locally sparse neural networks for tabular biomed- ical data. InInternational Conference on Machine Learning, pages 25123–25153. PMLR, 2022. 2

    2022

  35. [35]

    Language in a bottle: Language model guided con- cept bottlenecks for interpretable image classifica- tion

    Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in a bottle: Language model guided con- cept bottlenecks for interpretable image classifica- tion. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19187–19197, 2023. 4, 6, 8, 16

    2023

  36. [36]

    Post-hoc concept bottleneck models

    Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. InThe Eleventh International Conference on Learning Representa- tions, 2023. 2, 3, 4, 6, 7, 8, 9, 16

    2023

  37. [37]

    Correct-n- contrast: A contrastive approach for improving ro- bustness to spurious correlations.arXiv preprint arXiv:2203.01517, 2022

    Michael Zhang, Nimit S Sohoni, Hongyang R Zhang, Chelsea Finn, and Christopher R´ e. Correct-n- contrast: A contrastive approach for improving ro- bustness to spurious correlations.arXiv preprint arXiv:2203.01517, 2022. 3, 4, 6, 8, 16

    Pith/arXiv arXiv 2022

  38. [38]

    Places: A 10 million image database for scene recognition

    Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. InIEEE Trans- actions on Pattern Analysis and Machine Intelligence (TPAMI), volume 40, pages 1452–1464. IEEE, 2017. 6 11 A Technical Details of SEG- MIL-CBM This appendix provides the full technical details of the SEG-MI...

    2017

  39. [39]

    Classification Loss: Standard cross-entropy loss for predicting the correct image-level label

  40. [40]

    easy” and “hard

    Concept Alignment Loss: A cosine similar- ity loss between predicted concept activations ˆzi (normalized) and CLIP-derived similarity vectors zCLIP i : Lconcept =− 1 B BX i=1 cos ˆzi,z CLIP i , where B is the batch size (not to be confused with the bagB). The total loss is defined as: Ltotal =L cls +λ concept · Lconcept, where λconcept balances classifica...

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.