REVIEW 4 major objections 4 minor 21 references
Composing VLM-selected clinical tools with a learned bottleneck model yields medical-image classifiers that are accurate, interpretable, and data-efficient.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
A 'tool bottleneck' framework—VLM tool selection plus learned spatial fusion—matches or beats black-box classifiers, especially on scarce data.
T0 review reviewed 2026-08-03 challenge →
load-bearing objection The Tool Bottleneck idea is real and the Camelyon17 results are credible, but the dermatology numbers lean on ground-truth annotations passed as 'tools,' and the paper's broad claims do not survive that. the 4 major comments →
A Tool Bottleneck Framework for Clinically-Informed and Interpretable Medical Image Understanding
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The central claim is that composing a VLM-selected set of clinical tools through a learned bottleneck is a better inductive bias for medical image understanding than end-to-end learning or text-based tool composition. Concretely, the Tool Bottleneck Model rasterizes each tool's output into spatial maps, concatenates them, and feeds them to a CNN that outputs the prediction; tool knockout augmentation during training lets the model handle any VLM selection. Results on histopathology tumor detection and two dermatology classification tasks show accuracy/AUC on par or better than a standard CNN (e.g., 92.3% vs 88.6% on the histopathology task) and clear gains when only a few dozen labeled examp
What carries the argument
The Tool Bottleneck Model (TBM) is the central object: a frozen toolbox of clinically relevant tools produces feature maps, the TBM concatenates them across channels and applies a CNN to fuse them into an image-level prediction. Tool knockout augmentation—replacing unselected tools' outputs with a constant placeholder during training—makes the TBM robust to arbitrary tool selections and simultaneously learns the full conditional and every marginal conditional of the label given tool subsets.
Load-bearing premise
For the dermatology tasks, four of the seven tool channels are not produced by real tools but by ground-truth dermoscopic annotations treated as hypothetical detector outputs; if such detectors are unavailable or inaccurate in practice, the reported dermatology results are not established.
What would settle it
Run the dermatology experiments with the four dermoscopic structure maps produced by detectors trained on separate data (instead of ground-truth annotations) and compare AUC; if the framework's advantage over a standard CNN vanishes or reverses, the central claim fails for dermatology.
If this is right
- If TBF is correct, learned fusion of VLM-selected clinical tools is a viable alternative to end-to-end learning in medical imaging.
- TBF's data efficiency means small clinical datasets (tens of images) can still yield usable classifiers if a relevant toolbox exists.
- Tool-importance and intervention analyses give clinicians a way to inspect and correct model reasoning.
- Text-based tool composition (code or natural language) is the limiting factor for medical tool-use frameworks.
- Including all modality tools rather than VLM selection hurts performance, so tool selection matters.
Where Pith is reading between the lines
- The dermatology results rely in part on ground-truth structure annotations as hypothetical tool outputs; real deployment would require accurate detectors for pigment networks, streaks, etc., so the reported dermatology gains may reflect an upper bound.
- The framework's value should extend beyond classification to other medical tasks with spatially localized features (e.g., segmentation, grading), if suitable tools exist.
- A testable extension: use TBF with self-supervised or foundation-model tools to see whether the data-efficiency and interpretability benefits persist when tools are not domain-specific.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes the Tool Bottleneck Framework (TBF), which uses a medical VLM to select clinically relevant tools from a toolbox and then fuses the selected tools' outputs with a learned Tool Bottleneck Model (TBM) to make image-level predictions. TBF is evaluated on Camelyon17 (histopathology) and two ISIC 2017 binary tasks (dermatology), with reported accuracy/AUC on par with or better than CNNs and VLMs, plus data-efficiency and interpretability analyses. The method contribution is a learned composition mechanism for pixel-level and scalar tool outputs, enabled by a knockout-style training strategy that makes the TBM robust to arbitrary tool selections.
Significance. If the claims held, TBF would be a useful step toward clinically grounded and interpretable medical image classification: it generalizes Concept Bottleneck Models to spatially localized features, replaces text-based tool composition with a learned fusion network, and provides a simple strategy for handling arbitrary VLM tool selections. The authors are transparent in the appendix about the idealized nature of several dermatology tools, and the knockout proof in Appendix E is a sound adaptation of prior work. However, the headline claim—that TBF performs 'on par with or better than' black-box models and tool-use frameworks across histopathology and dermatology—is only cleanly supported on Camelyon17. Two of the three evaluation tasks rely on ground-truth dermoscopic structure annotations as 'hypothetical tool outputs' at test time, which is an oracle setup, not a realized tool-use system.
major comments (4)
- [Appendix C.2, Table 1, Sections 4.1 and 5.1] Four of the seven ISIC tools (pigment network, negative network, streaks, milia-like cysts) are not produced by learned detectors at test time; they are ground-truth dermoscopic feature annotations from the ISIC 2017 dataset. Appendix C.2 states this explicitly, calling it 'an idealized scenario in which accurate detectors ... 'exist''. These annotations are label-adjacent: milia-like cysts are strongly indicative of seborrheic keratosis (the non-melanocytic class in ISIC-MN), and pigment network is associated with melanocytic lesions. The TBM therefore receives structured label-adjacent information that the EfficientNet baseline does not. Since two of the three main tasks (ISIC-BM, ISIC-MN) and the data-efficiency experiment in Figure 2(b) depend on these oracle maps, the central claim of 'on par with or better than ... across histopathology and dermatology' is unsupported for a realize
- [Figure 2(b), Section 5.2] The data-efficiency argument is a key contribution claim, but the ISIC-MN panel is compromised by the same oracle-tool setup. The reported gains of TBF over EfficientNet in the 4–64 image regime may simply reflect that the TBM is fed ground-truth dermoscopic structure maps, whereas EfficientNet receives only raw pixels. These low-data results cannot be interpreted as evidence that the TBF framework, with realistically available tools, is more data-efficient. The claim 'TBF embodies clinically-relevant inductive biases' therefore needs to be re-established on Camelyon17 alone, or with predicted (non-oracle) ISIC tools. As written, the ISIC-MN data-efficiency result is an upper-bound experiment, not a demonstration of the framework.
- [Appendix D.1, Table 2 vs. Table 1] There is an internal inconsistency in the reported Camelyon17 accuracy for TBF with ImageNet pretraining: Table 1 reports 92.3%, while Table 2 reports 92.8%, and the text in D.1 says the accuracy 'increases to 92.3%'. The discrepancy needs to be resolved and the correct value reported consistently. If the difference is due to different evaluation splits or seeds, that should be stated.
- [Appendix B, Y-Net baseline] The paper repeatedly states that Y-Net is trained on 'the same amount and types of data' as TBF (Sections 4.3, 5.1, Appendix B). However, TBF receives five histopathology tool maps or seven dermatology tool maps, while Y-Net receives only a single segmentation mask (tumor masks for Camelyon, lesion masks for ISIC) plus raw images. This is not the same type or amount of pixel-level supervision. The comparison is still meaningful as a segmentation-for-classification baseline, but the 'same data' claim should be softened or the Y-Net training should be updated to consume the full tool-stack inputs.
minor comments (4)
- [Section 5.1, Table 1] The accuracy/AUC values are reported without confidence intervals or significance tests. Given the small differences in several comparisons (e.g., ISIC-MN 91.8 vs. 91.2), error bars or a significance statement would help the reader judge whether TBF is truly 'on par or better'.
- [Section 3.3, Equation (1)] The Bernoulli perturbation with parameter p = (1−α)0.5 + α s_i is described in words, but the notation is not fully formalized. Specifying the exact sampling distribution and how the placeholder value ¯z_i is applied would improve reproducibility.
- [Appendix C.2] The appendix uses the phrase 'hypothetical tool outputs' once, but the discussion of the four ground-truth structure maps would be clearer if the authors consistently labeled them as oracle inputs throughout the paper, including in the figure captions and table notes. Currently, Table 1 does not indicate that any ISIC results use ground-truth annotations.
- [Appendix D.1] The sentence 'TBF achieves 86.7% accuracy when trained from scratch, and increases to 92.3%' is inconsistent with Table 2, which lists 92.8% for the ImageNet-pretrained TBF. This is likely a typo, but it should be corrected.
Circularity Check
No equation-level circularity: the TBM training and knockout derivations are self-contained. The main caveat is an oracle-tool evaluation issue on ISIC, not a circular-derivation defect.
full rationale
The paper's derivation chain is not circular. The Tool Bottleneck Model is trained on frozen tool outputs: tools are fixed, the bottleneck tensor is the concatenation of tool maps, and f_theta is a CNN trained with cross-entropy or binary cross-entropy. The tool-knockout argument in Appendix E is an explicit adaptation of Nguyen et al. (2025), with the independence-of-mask derivation given in the paper itself; it does not presuppose the result. Cited prior work by the authors (Hsu et al., 2023, 2025) is contextual and not load-bearing; no uniqueness theorem or forced ansatz is imported from it. The central issue in the paper is the ISIC evaluation: Appendix C.2 states plainly that four of seven dermatology tool channels are 'ground-truth dermoscopic feature annotations provided by ISIC 2017 as hypothetical tool outputs—representing an idealized scenario in which accurate detectors for these clinically meaningful cues exist.' This means the reported ISIC-BM and ISIC-MN results, and the ISIC-MN data-efficiency curves, are an oracle upper bound rather than evidence about real tool use. That is a soundness and data-integrity limitation, not a circular derivation: the final label is not defined by the tool maps in the paper, and the model does not rename the tool outputs as predictions. The Camelyon17 experiments use predicted HoVer-Net outputs and do not share this caveat. Because the core TBM derivation is self-contained and the only serious concern is an oracle-evaluation caveat, the circularity score is low.
Axiom & Free-Parameter Ledger
free parameters (3)
- alpha (Bernoulli perturbation strength) =
0.9 (Camelyon17, ISIC-BM), 0.8 (ISIC-MN)
- k (number of VLM-selected tools) =
3 for all tasks
- Handcrafted color-marker thresholds
axioms (4)
- domain assumption The VLM (MedGemma) produces useful tool selections for each image and task.
- domain assumption All tools in the toolbox are available as real detectors at test time and capture clinically relevant features.
- standard math Tool knockout augmentation lets f_theta learn all marginal conditionals p(Y | Z_{-M}).
- domain assumption HoVer-Net nucleus predictions transfer to Camelyon17 patches without fine-tuning.
Cite this review
Pith. "Pith review of A Tool Bottleneck Framework for Clinically-Informed and Interpretable Medical Image Understanding." pith.science (2026). https://pith.science/paper/5HUVFJON
@misc{pith2026251221414,
author = {Pith},
title = {Pith review of: A Tool Bottleneck Framework for Clinically-Informed and Interpretable Medical Image Understanding},
year = {2026},
howpublished = {\url{https://pith.science/paper/5HUVFJON}},
note = {Machine review of arXiv:2512.21414}
}
read the original abstract
Recent tool-use frameworks powered by vision-language models (VLMs) improve image understanding by grounding model predictions with specialized tools. Broadly, these frameworks leverage VLMs and a pre-specified toolbox to decompose the prediction task into multiple tool calls (often deep learning models) which are composed to make a prediction. The dominant approach to composing tools is using text, via function calls embedded in VLM-generated code or natural language. However, these methods often perform poorly on medical image understanding, where salient information is encoded as spatially-localized features that are difficult to compose or fuse via text alone. To address this, we propose a tool-use framework for medical image understanding called the Tool Bottleneck Framework (TBF), which composes VLM-selected tools using a learned Tool Bottleneck Model (TBM). For a given image and task, TBF leverages an off-the-shelf medical VLM to select tools from a toolbox that each extract clinically-relevant features. Instead of text-based composition, these tools are composed by the TBM, which computes and fuses the tool outputs using a neural network before outputting the final prediction. We propose a simple and effective strategy for TBMs to make predictions with any arbitrary VLM tool selection. Overall, our framework not only improves tool-use in medical imaging contexts, but also yields more interpretable, clinically-grounded predictors. We evaluate TBF on tasks in histopathology and dermatology and find that these advantages enable our framework to perform on par with or better than deep learning-based classifiers, VLMs, and state-of-the-art tool-use frameworks, with particular gains in data-limited regimes. The project details and the code are available at https://christinaliu2020.github.io/tbm/.
Figures
Reference graph
Works this paper leans on
-
[1]
h i s t o _ n u c _ c e n t r o i d
As mentioned in the main paper, this is easily implemented by replacing the tool outputzi with¯zi of the same shape. 24 Tool Bottleneck Framework To see thatf θ learns all such marginals simultaneously, consider the expected training loss under tool knockout: L(θ) =E Z′,Y EM ℓ Y, fθ(Z′(M,Z) (4) =E Z,Y EM X m∈M I(M=m)ℓ Y, fθ(Z′(m,Z)) (5) =E Z,Y X m∈M p(M=m...
2017
-
[3]
21 Liu W ang Hsu Wu Adeli Table 3: Performance of TBF ablations across Camelyon17 (Accuracy) and ISIC 2017 (AUC). Model Camelyon17 ISIC-BM ISIC-MN Bernoulli 90.3 72.7 90.1 α= 0.5 90.5 75.3 88.9 α= 0.6 91.3 75.1 89.1 α= 0.7 90.7 75.6 91.2 α= 0.8 91.6 75.3 91.9 α= 0.992.3 77.090.6 α= 1.0 90.2 74.9 90.4 Random top-k91.7 76.192.0 Dynamic top-k91.4 66.5 90.4 A...
2017
-
[4]
missing” value of−1 (a constant map of −1s) and are handled consistently across all masking regimes. Details on filling dropped tools with the “missing
As described in Section 4.1, all Camelyon17 tool maps are scaled to lie in [0,1] before masking; dropped tools are represented by channels filled with a constant “missing” value of−1 (a constant map of −1s) and are handled consistently across all masking regimes. Details on filling dropped tools with the “missing” values are provided in Section E. C.2. IS...
2017
-
[6]
Appendix E. Details on Tool Knockout We show that our tool knockout augmentation enablesf θ to estimate the full conditional (the distribution ofY=yconditioned on all tool outputs) and all marginal conditionals (the distributionY=yconditioned on any subset of tool outputs). Our argument follows the theoretical analysis of Knockout by Nguyen et al. (Nguyen...
2025
-
[9]
Correlation of nuclear morphometry with progres- sion of breast cancer.Cancer, 68(9):2012–2016,
Kenneth J Pienta and Donald S Coffey. Correlation of nuclear morphometry with progres- sion of breast cancer.Cancer, 68(9):2012–2016,
2012
-
[10]
Medgemma technical report.arXiv preprint arXiv:2507.05201,
Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, C ´ ıan Hughes, Charles Lau, et al. Medgemma technical report.arXiv preprint arXiv:2507.05201,
-
[11]
Medclip: Contrastive learning from unpaired medical images and text
Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medclip: Contrastive learning from unpaired medical images and text. InProceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, volume 2022, page 3876,
2022
-
[13]
An Yan, Yu Wang, Yiwu Zhong, Zexue He, Petros Karypis, Zihan Wang, Chengyu Dong, Amilcare Gentili, Chun-Nan Hsu, Jingbo Shang, et al. Robust and interpretable medical image classifiers via concept bottleneck models.arXiv preprint arXiv:2310.03182,
-
[14]
Each WSI is manually annotated by pathologists to mark tumor regions, from which non-overlapping 96×96 pixel patches are extracted and labeled as eitherTumororNormal
is adapted from the CAME- LYON17 challenge (Litjens et al., 2018), which consists of whole-slide images (WSIs) of breast cancer metastases in lymph node sections. Each WSI is manually annotated by pathologists to mark tumor regions, from which non-overlapping 96×96 pixel patches are extracted and labeled as eitherTumororNormal. A patch is labeledTu- morif...
2018
-
[16]
tumor”, “no tumor
for 25 epochs, and monitored with Dice score. In Stage 2, we jointly train the segmentation and classification head with the combined lossL=L seg +L cls,where bothL seg andL cls are cross-entropy losses for the segmentation mask and binary label, respectively. Stage 2 uses Adam with a lower learning rate (3×10 −4), batch size 16, and 20 epochs. For both d...
2023
-
[19]
22 Tool Bottleneck Framework Figure 5: Distribution of MedGemma selected tool combinations for Camelyon17 D.4. T ool Output Intervention Besides tool importance discussed in Section 5.3, another method for interrogating TBF’s decision-making is to intervene or manipulate the tool outputs, since the features encoded in the tool outputs are clinically meani...
2013
-
[1984]
12 Tool Bottleneck Framework Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tris- tan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language- and-vision assistant for biomedicine in one day.arXiv preprint arXiv:2306.00890,
-
[2003]
Noel CF Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, et al. Skin lesion analysis toward melanoma detection: A challenge at the 2017 interna- tional symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isi...
2017
-
[2012]
shades of gray
and color-constancy-based preprocessing for robust skin image analysis (Ciurea and Funt, 2003). Following these approaches, we first apply a simple “shades of gray” color con- stancy transform (Ciurea and Funt,
2003
-
[2016]
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, To- bias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications.arXiv preprint arXiv:1704.04861,
-
[2017]
Probabilistic concept bottleneck models.arXiv preprint arXiv:2306.01574, 2023a
Eunji Kim, Dahuin Jung, Sangha Park, Siwon Kim, and Sungroh Yoon. Probabilistic concept bottleneck models.arXiv preprint arXiv:2306.01574, 2023a. Injae Kim, Jongha Kim, Joonmyung Choi, and Hyunwoo J. Kim. Concept bottleneck with visual concept filtering for explainable medical image classification.arXiv preprint arXiv:2308.11920, 2023b. Accepted to MedAGI...
-
[2018]
X Liu, J Wu, X Luo, W Liao, S Zhang, and G Wang
ISSN 2047-217X. X Liu, J Wu, X Luo, W Liao, S Zhang, and G Wang. 2024 ieee international symposium on biomedical imaging (isbi).IEEE International Symposium on Biomedical Imaging (ISBI),
2047
-
[2021]
Yi Lu, Jiawang Cao, Yongliang Wu, Bozheng Li, Licheng Tang, Yangguang Ji, Chong Wu, Jay Wu, and Wenbo Zhu. Rsvp: Reasoning segmentation via visual prompting and multi-modal chain-of-thought.arXiv preprint arXiv:2506.04277,
-
[2022]
Graph concept bottleneck models.arXiv preprint arXiv:2508.14255,
Haotian Xu, Tsui-Wei Weng, Lam M Nguyen, and Tengfei Ma. Graph concept bottleneck models.arXiv preprint arXiv:2508.14255,
-
[2023]
Integrating Clinical Knowl- edge into Concept Bottleneck Models
13 Liu W ang Hsu Wu Adeli Winnie Pang, Xueyi Ke, Satoshi Tsutsui, and Bihan Wen. Integrating Clinical Knowl- edge into Concept Bottleneck Models. InProceedings of Medical Image Computing and Computer Assisted Intervention – MICCAI 2024, volume LNCS 15004, pages 243–253. Springer Nature Switzerland, October
2024
-
[2025]
Label-free concept bottleneck models.arXiv preprint arXiv:2304.06129,
Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models.arXiv preprint arXiv:2304.06129,
This paper was first reviewed by deepseek-v4-flash on August 3, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.