{"id":"b7de5253-d906-4ac6-9fe9-08890a1acb75","arxiv_id":"2505.07209","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"DOT-CBM uses optimal transport between image patches and concept embeddings, with disentanglement and bias priors, to improve accuracy and localize concepts.","lead":"This paper introduces DOT-CBM, a concept bottleneck model that aligns image patches with human-defined concepts using optimal transport, then predicts classes from the resulting concept scores. A reader interested in interpretable AI will see a method that yields both competitive classification and explicit heatmaps of where each concept appears.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation 18's concept prior γ = 1 − Softmax(r) sums to M−1, not 1, making the stated Sinkhorn problem infeasible and Q* undefined; this undermines the OT-based alignment until corrected.","rationale":"I agree with the reader's weakest-assumption analysis: Eq. 18 is genuinely invalid as a probability distribution, and the infeasibility of the stated OT constraints is load-bearing because the entire method—concept activations, inversion masks, and the associated losses—depends on Q*. The concern is concrete and can be settled by a simple summation check or by inspecting the code for an unstated normalization. This is not an objection to the general idea; renormalizing γ would likely make the formulation coherent, which is why I do not recommend moving from the reader's CONDITIONAL verdict to REJECT. I also considered whether the activation rule in Eq. 12 (taking the minimum weighted cost per concept) is a deeper problem, since it is heuristic and not a standard OT aggregation, but it is at least a well-defined design choice once Q* exists. The gamma flaw is more fundamental because it makes Q* undefined as written. No other passage in the manuscript provides a missing proof or limitation statement that changes this assessment. Therefore the reader's CONDITIONAL verdict stands, with the requested revision being a corrected, explicitly normalized formulation of Eq. 18 and ideally released code confirming the exact implementation.","tokens_in":14064,"tokens_out":4579,"duration_ms":48044,"concrete_test":"Compute the sum of γ from Eq. 18 on any real dataset, e.g., CUB with M = 112 concepts. If the sum equals M−1, inspect the released training code to see whether γ is L1-normalized before being passed to Sinkhorn. If no normalization appears, run one Sinkhorn iteration with θ = Softmax(SI) and γ as in Eq. 18 and check whether the column marginal Q^T 1_N actually equals γ and whether the total mass is 1; the mismatch will confirm infeasibility. If normalization does appear, update Eq. 18 to state it explicitly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The weakest link is the concept-set prior γ defined in Eq. 18. The text defines r_i as a scalar conditional co-occurrence sum in Eq. 17 and then sets γ = 1 − Softmax(r_i). Since Softmax(r) is a probability vector whose entries sum to 1, the vector 1 − Softmax(r) has entries summing to M−1, not 1. Equation 9 requires Q 1_M = θ and Q^T 1_N = γ, where θ sums to 1. The total mass of any feasible Q must then equal both 1 (from the row-sum constraint) and M−1 (from the column-sum constraint). For M > 2 this is impossible, so the Sinkhorn problem in Eq. 9–10 has no feasible solution and Q* is undefined unless the vector is silently renormalized somewhere not stated in the paper. All downstream quantities—concept activations (Eq. 12), inversion masks (Eq. 13), and the concept loss (Eq. 14)—are computed from Q*, so this is not a cosmetic typo but a formal gap in the method as written. The paper neither states a normalization nor releases code to show one, so the central fine-grained alignment claim is not reproducible from the mathematical description.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DOT-CBM, a Concept Bottleneck Model that models the image-to-concept mapping as an optimal transport problem between local image patches (from DINOv2) and concept embeddings (from a CLIP text encoder). Two orthogonality losses are used to disentangle patch features and concept features, and two priors (a saliency-based patch prior and a co-occurrence-based concept prior) are introduced to reduce spurious visual-concept correlations. The learned transport plan Q* yields concept activations through a minimum-product rule and inversion masks through its columns. The model is trained in two stages and evaluated on image classification (ImageNet, CUB, CIFAR-100, AwA2), part detection (PartImageNet, CUB, RIVAL-10, PASCAL-Parts), and out-of-distribution generalization, reporting state-of-the-art results against re-implemented baselines. Ablations and qualitative visualizations are also provided.","tokens_in":14376,"tokens_out":4206,"duration_ms":41848,"significance":"If the technical issues are resolved, the paper addresses a real limitation of existing CBMs—the lack of explicit fine-grained alignment between image regions and concepts—and offers a natural mechanism for visualizing concept predictions. The reported gains on part detection and OOD generalization are substantial, and the inclusion of dual priors to counter dataset bias is a thoughtful design. The manuscript also credibly reimplements baselines with a controlled backbone and concept set. However, the central derivation contains a formal gap (Eq. 18) and the concept activation rule (Eq. 12) is not justified, which currently prevents the acceptance of the method as described. The absence of error bars and code further weakens the empirical claims.","major_comments":[{"comment":"The definition γ = 1 − Softmax(r_i) produces a vector whose entries sum to M − 1, not 1, when r_i is a scalar per concept and M > 2. This violates the simplex constraint on γ and makes the column marginal Q^T 1_N = γ in Eq. (9) infeasible. As a result, the Sinkhorn problem in Eqs. (9)–(10) has no feasible solution, and the optimal Q* used in Eqs. (11)–(14) is undefined as written. The paper must explicitly renormalize the vector, define a different mapping (e.g., 1 − Softmax(r_i) normalized by M−1, or a softmax over the negative co-occurrence counts), or otherwise state the missing normalization step. This is a load-bearing issue because all downstream quantities—concept activations, inversion masks, and the concept loss—depend on Q*.","section":"Section 3.4, Eq. (18)"},{"comment":"The concept activation rule â_j = σ(1 − min_i(q_ij d_ij)) is introduced without derivation or empirical justification. The standard OT cost per concept would be Σ_i q_ij d_ij or an aggregate transport quantity; the minimum of products is not a natural consequence of the OT formulation. The paper should either derive this rule from the OT objective, motivate it as a design choice with reference to competing alternatives, or include an ablation study comparing min, mean, and sum aggregations. Without this, the connection between the OT plan and the reported concept prediction quality is not established.","section":"Section 3.3, Eq. (12)"},{"comment":"The inversion mask for concept c_j is defined directly as the column vector m_j = [q_1j, ..., q_Nj] (Eq. 13), and the same Q* is used to compute concept activations (Eq. 12) and to produce part detection masks. Consequently, part detection mAP partially measures how well the training objective (concept-label BCE loss) has been optimized, rather than providing independent evidence of fine-grained alignment. The classification and OOD experiments are external, but the part detection claim is a central contribution. I recommend validating the masks on held-out concepts or images, or comparing with an OT plan that is not supervised by concept labels, to demonstrate that the alignment is genuinely meaningful.","section":"Section 4.3, part detection evaluation"},{"comment":"All tables (Tables 1, 2, 3) report a single run without standard deviations, error bars, or significance tests. The reported margins are often small (e.g., +0.91 on ImageNet, −0.25 on AwA2), so the paper should include multiple seeds and confidence intervals to support the claimed improvements. Additionally, the manuscript does not provide a code link or any other reproduction artifact; given the formal gap in Eq. (18), releasing code would be particularly important to confirm the method as described.","section":"Section 4, experimental reporting"}],"minor_comments":[{"comment":"The text refers to Q as a 'doubly stochastic matrix,' but with general marginals θ and γ the feasible set is a transportation polytope, not a doubly stochastic matrix. Please correct the terminology to avoid confusion with the standard definition.","section":"Section 3.3, Eq. (9)"},{"comment":"The phrase 'sum= Softmax(1− )' in the figure caption appears garbled and should be rewritten to clearly describe the prior computation for the concept set.","section":"Figure 2 caption"},{"comment":"The hyperparameters λ1, λ2, λ3 are specified only for CUB; the appendix should state the values used for all datasets, or explicitly confirm that the same values are used everywhere.","section":"Section 4.2, hyperparameters"},{"comment":"The abbreviation 'LrD' is used in the table but is not defined in the caption; it is only explained in the body text of Section 4.5. Please define it in the caption or at first use in the table.","section":"Table 3 caption"},{"comment":"The OOD experiment relies on SAM-generated segmentation masks for the foreground/background split; please report a validation of the mask quality (e.g., IoU against ground truth, if any) so that the experimental setup is fully transparent.","section":"Section 4.4, OOD experiment"},{"comment":"Some references are incomplete (e.g., entries for arXiv preprints lack version numbers or access dates). Please ensure all bibliographic entries are complete and consistent.","section":"General presentation"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a conceptually interesting extension of CBMs, and the experimental coverage is broad. The central issue is the formal infeasibility of the OT problem caused by Eq. (18), which must be corrected before the method can be considered defined. The activation rule in Eq. (12) also needs a justification or ablation. Given the magnitude of the reported improvements on part detection and OOD generalization, I believe the paper has promise, but the current version is not reproducible from the mathematical description and lacks statistical rigor. I recommend major revision rather than rejection, provided the authors can address these points. I would also encourage the editor to consider that the paper does not cite a few closely related OT-based alignment works beyond PatchCT; however, the proposed method appears sufficiently distinct in its use of priors and disentanglement."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a plausible and fairly new combination — using optimal transport to align image patches with concept embeddings inside a concept bottleneck model, with orthogonal projection losses and two data-driven priors. The inversion heatmaps are a genuinely useful byproduct, and the OOD background-shift experiment is a nice touch. On the merits, the idea is worth engaging with.\n\nThe good parts: the OT framing is a natural extension of PatchCT to the CBM setting, and the authors do a fair job of situating it against recent CBM variants. The ablation table is internally consistent: each component helps, and the component-wise improvements line up with the narrative. The part-detection gains over Grad-CAM baselines are large, and the qualitative results support the claim that the inversion masks are more localized. I also appreciate that they re-implement all baselines with the same backbone and concept set — that is the right way to compare, even if it needs careful checking.\n\nThe soft spots: the formal gap in Eq. 18 is not cosmetic. As written, γ = 1 − Softmax(r_i) sums to M−1, not 1, so the column marginal Q^T 1_N = γ is inconsistent with the row marginal Q 1_M = θ. For M > 2, there is no feasible Q, and the Sinkhorn problem is undefined. The paper does not state a renormalization, and no code is released to silently fix it. Since Q* is used to derive the concept activations, the inversion masks, and the concept loss, this breaks the method description as written. The fix is probably simple — normalize γ or use Softmax(1 − r_i) — but it must be stated.\n\nThe min_i(q_ij d_ij) activation rule is another heuristic that gets no derivation; it is plausible but the reader should be told why min is the right aggregation. All tables lack error bars, which matters when the claimed gains over CoopCBM on ImageNet and CUB are under 1 and 3.3 points respectively. On the circularity concern: the part-detection evaluation does use the same Q* that supervises the concept loss, so part of that result is a check of the training objective rather than fully independent. The classification and OOD numbers are external benchmarks and give the paper independent value.\n\nBottom line: the central idea is defensible, the experiments are broad, and the formal bug is very likely fixable. This paper deserves a serious referee — the reviewer should push for the normalization, code release, and error bars, but not desk-reject the idea. I would bring it to reading group once a revised version with the corrected gamma is available, and I would not cite it in its current arXiv form until that fix is explicit.","headline":"A genuinely interesting CBM extension that marries OT-based patch-concept alignment to concept bottlenecks, but Eq. 18's concept prior is formally broken as written and the method section needs a fix before the SOTA claims can be taken at face value.","tokens_in":14873,"tokens_out":1674,"would_cite":false,"duration_ms":18639,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Modeling concept prediction as optimal transport between image patches and concept embeddings yields explicit visual-concept alignment, more reliable concept predictions, and state-of-the-art classification, part detection, and OOD…","keywords":["concept bottleneck models","optimal transport","fine-grained visual-concept alignment","interpretability","spurious correlation","part detection","out-of-distribution generalization","disentanglement"],"falsifier":"On any dataset used in the paper, compute $\\sum_{j=1}^M \\gamma_j$ from Eq. (18) with $M$ the number of concepts; it equals $M-1$, not $1$. If the released training code is run with $\\gamma$ exactly as stated, the Sinkhorn iteration cannot satisfy the column marginal $Q^\\top \\mathbf{1}_N = \\gamma$, so the concept activations and inversion masks would either fail or depend on an invisible normalization; changing the normalization rule should measurably change every reported mask and accuracy.","tokens_in":13863,"feed_emoji":"🎯","tokens_out":11374,"duration_ms":96932,"temperature":0.7,"pith_summary":"This paper tries to establish that a Concept Bottleneck Model (an interpretable classifier that predicts human-understandable concepts before the final class) can align those concepts to specific image regions by treating concept prediction as an optimal transport problem—a matching that moves mass from local image patches to concept embeddings. The payoff is a single transport plan that serves both as the concept predictor and as an explicit inversion heatmap, so users can see which pixels caused each concept prediction. The paper reports that this design outperforms prior concept models on ImageNet (83.84%), CUB (85.39%), and part detection (50.12 mAP on PartImageNet), and improves out-of-distribution accuracy on background-shifted datasets. The takeaway is that fine-grained, explicit alignment plus two data-bias priors can make concept models both more accurate and more interpretable.","feed_headline":"Optimal transport links image patches to concepts, beating prior models","feed_subtitle":"Fine-grained alignment lifts classification, part detection, and out-of-distribution robustness","key_machinery":"The carrying object is the optimal transport plan $Q^*$, an $N \\times M$ matrix coupling image patches to concept embeddings, computed by Sinkhorn fixed-point iterations on a cosine-distance cost with prescribed row and column sums $\\theta$ and $\\gamma$. The same matrix is read in two directions: each concept activation $\\hat{a}_j$ comes from $1 - \\min_i (q^*_{ij} d_{ij})$ after a sigmoid, and each concept's inversion mask is the column vector $(q^*_{1j}, \\dots, q^*_{Nj})$. Two orthogonal projection losses, one on patch features and one on concept features, push each modality onto a unit hypersphere with low cosine similarity, while the saliency and co-occurrence priors shape the marginals. The Sinkhorn plan is what makes the image-to-concept map no longer a black box: it is an explicit, invertible assignment rather than a learned global feature.","core_discovery":"The central claim is that replacing the black-box mapping from an entire image to concept labels with an optimal transport problem between local image patches and concept features eliminates the coarse-grained, spurious visual-concept associations typical of vanilla Concept Bottleneck Models. After orthogonal-projection losses disentangle patch features and concept features, the Sinkhorn solution $Q^*$—an $N \\times M$ assignment matrix—defines every downstream quantity: concept activations are computed from the minimum transport distance between a concept and all patches, and the concept inversion mask is the corresponding column of $Q^*$. Two prior distributions (a saliency-based patch prior and a conditional co-occurrence concept prior) penalize shortcut alignments to background and to frequently co-occurring concepts. The paper's evidence is that this single mechanism simultaneously raises classification accuracy, part-detection mAP by roughly eight points over the best reimplemented baseline, and out-of-distribution generalization under background shift.","pith_inferences":["A natural extension the paper does not pursue is interactive intervention: because $Q^*$ is explicit, a user could manually edit the alignment or point to a patch and ask which concepts it feeds, turning the inversion mask into a debugging tool rather than just a visualization.","The reported OOD gains likely depend on the quality of the saliency map from the pretrained vision transformer; an ablation that replaces the fixed saliency prior with a self-refined mask would show how much robustness comes from the prior versus from the OT alignment itself.","The concept-set prior in Eq. (18) is written as $1 - \\mathrm{Softmax}(r_i)$, which sums to $M-1$, not $1$; if an unstated renormalization is applied in code, its exact form is an underdocumented degree of freedom that changes the Sinkhorn marginals and therefore every inversion mask.","The same patch-to-concept transport could transfer to other bottleneck tasks where part-level attribution matters, such as medical image diagnosis, by swapping the concept bank without changing the alignment machinery."],"forward_implications":["Concept predictions no longer need post-hoc localization: the inversion mask is a free byproduct of the transport plan, and the paper measures it directly with part-detection mAP.","Fine-grained patch-concept alignment reduces part-background, part-object, and part-part spurious correlations, which the paper links to classification gains of 0.91 points on ImageNet and 3.29 points on CUB over the best reimplemented baselines.","Saliency-based and co-occurrence-based priors improve out-of-distribution accuracy sharply under background shift: +10.3 on CUB and +10.7 on Dogs over the strongest baseline.","Because the concept-to-class layer spontaneously becomes sparse under the orthogonal losses, the model retains a human-readable concept-to-class explanation while approaching black-box accuracy.","The framework is compatible with frozen pretrained encoders, so the same recipe can be applied at ImageNet scale without heavy training compute."],"supporting_citations":[{"why":"Introduces the Concept Bottleneck Model and the two-stage independent training that DOT-CBM builds on and compares against as the vanilla baseline.","marker":"[17]"},{"why":"Supplies Sinkhorn fixed-point iterations, the algorithm used to compute the optimal transport plan.","marker":"[7]"},{"why":"Provides the pretrained vision transformer whose patch features and saliency map feed the visual side of the transport problem.","marker":"[32]"},{"why":"Provides the pretrained text encoder that produces the concept embeddings aligned with image patches.","marker":"[33]"},{"why":"Supplies the high-quality concept bank used to re-implement all baselines and DOT-CBM for fair comparison.","marker":"[47]"},{"why":"The strongest reimplemented baseline; its auxiliary orthogonality loss and its out-of-distribution background-shift protocol are the comparison points DOT-CBM extends.","marker":"[39]"},{"why":"Used to segment foreground objects for the out-of-distribution colored-background experiments that the OOD claims rest on.","marker":"[16]"},{"why":"Provides the PartImageNet part-segmentation benchmark whose mAP scores measure the accuracy of the inversion masks.","marker":"[13]"}],"fun_headline_variants":["Optimal transport aligns image patches to concepts for fine-grained interpretability","DOT-CBM: Transport-based concept bottlenecks boost part detection and OOD","Patch-concept optimal transport lifts reliability and part detection","Disentangled transport maps patches to concepts, sharpening interpretability","Patch-level optimal transport cuts spurious concept links and boosts OOD"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the concept-side prior is a true probability distribution adding up to one; as written it adds up to one fewer than the number of concepts, so without an unstated renormalization the matching problem has no solution and every reported result is undefined.","fun_headline_variants_meta":{"raw":{"variants":["Optimal transport aligns image patches to concepts for fine-grained interpretability","DOT-CBM: Transport-based concept bottlenecks boost part detection and OOD","Patch-concept optimal transport lifts reliability and part detection","Disentangled transport maps patches to concepts, sharpening interpretability","Patch-level optimal transport cuts spurious concept links and boosts OOD"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001112,"raw_usage":{"total_tokens":4646,"prompt_tokens":973,"completion_tokens":3673,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":3584}},"tokens_in":589,"tokens_out":3673,"duration_ms":23797,"temperature":1.0,"reasoning_tokens":3584,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:22:43.294081+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On any dataset used in the paper, compute $\\sum_{j=1}^M \\gamma_j$ from Eq. (18) with $M$ the number of concepts; it equals $M-1$, not $1$. If the released training code is run with $\\gamma$ exactly as stated, the Sinkhorn iteration cannot satisfy the column marginal $Q^\\top \\mathbf{1}_N = \\gamma$, so the concept activations and inversion masks would either fail or depend on an invisible normalization; changing the normalization rule should measurably change every reported mask and accuracy.","supporting_citations":[{"cited_title":"Concept bottleneck models","cited_arxiv_id":null,"evidence_quote":"Introduces the Concept Bottleneck Model and the two-stage independent training that DOT-CBM builds on and compares against as the vanilla baseline."},{"cited_title":"Sinkhorn distances: Lightspeed computation of optimal transport","cited_arxiv_id":null,"evidence_quote":"Supplies Sinkhorn fixed-point iterations, the algorithm used to compute the optimal transport plan."},{"cited_title":"Auxiliary losses for learning generalizable concept-based models","cited_arxiv_id":null,"evidence_quote":"The strongest reimplemented baseline; its auxiliary orthogonality loss and its out-of-distribution background-shift protocol are the comparison points DOT-CBM extends."},{"cited_title":"Partimagenet: A large, high- quality dataset of parts","cited_arxiv_id":null,"evidence_quote":"Provides the PartImageNet part-segmentation benchmark whose mAP scores measure the accuracy of the inversion masks."}],"review_version":1}