{"id":"f859c8d7-cbd3-436c-943b-95674db2f743","arxiv_id":"2512.21414","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A 'tool bottleneck' framework—VLM tool selection plus learned spatial fusion—matches or beats black-box classifiers, especially on scarce data.","lead":"This paper introduces a framework in which a medical vision-language model chooses which imaging tools to run, and a small learned network then combines the tool outputs into a diagnosis. It reports gains over standard classifiers on breast-cancer and skin-lesion benchmarks, with the largest advantages when only a few labeled images are available.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim leans on dermatology results where four of seven ISIC tool channels are ground-truth annotations, not predicted tool outputs; those oracle maps make the ISIC and data-efficiency numbers uninterpretable as tool-use evidence.","rationale":"The reader's weakest-assumption analysis identifies exactly the load-bearing flaw: the ISIC experiments use ground-truth structure annotations as tools, which is an oracle setup. I agree with that assessment. The central claim is not merely about Camelyon17; it is about a general framework for clinically-informed medical image understanding across histopathology and dermatology. Two of the three evaluated tasks are dermatology, and those two tasks rely on four ground-truth structure channels that encode the clinical cues being predicted. This makes the reported dermatology performance and the data-efficiency story an upper bound on an idealized scenario where perfect detectors exist, not a demonstration that the framework works with realistically available tools. The Camelyon17 result is a genuine positive: the tools are predicted by HoVer-Net, and the improvement over EfficientNet (92.3 vs 88.6) is meaningful. But one positive task cannot support the abstract's across-modality claim, especially when the headline numbers in dermatology are likely inflated by oracle access. A revised paper that replaces the oracle structure maps with predicted maps, reports error bars, and restricts its claims accordingly could be conditionally acceptable; as written, the central empirical claim is not established. The concern is not about honesty—the appendix does disclose the setup—but about validity of the conclusion drawn from that setup.","tokens_in":20380,"tokens_out":3053,"duration_ms":40134,"concrete_test":"Retrain TBF on ISIC-BM and ISIC-MN with the four dermoscopic structure channels produced by real detectors: e.g., train a segmentation model on the ISIC training superpixel labels for pigment network, negative network, streaks, and milia-like cysts, or use a public dermoscopic structure segmentation model, and feed those predicted maps to the TBM instead of the ground-truth maps. Then rerun Table 1 (ISIC rows) and the Figure 2b data-efficiency curves against EfficientNet. If the ISIC-MN AUC advantage and the 4-64 image low-data gains shrink or reverse, the oracle-tool assumption is the source; if they persist, the central claim is materially supported.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's headline claim is that TBF performs \"on par with or better than deep learning-based classifiers, VLMs, and state-of-the-art tool-use frameworks\" across histopathology and dermatology. That claim depends heavily on the ISIC-BM and ISIC-MN experiments, because two of the three main tasks are dermatology. Appendix C.2 states plainly: for the dermoscopic structure tools (pigment network, negative network, streaks, milia-like cysts), the authors \"use the ground-truth dermoscopic feature annotations provided by ISIC 2017 as hypothetical tool outputs—representing an idealized scenario in which accurate detectors for these clinically meaningful cues 'exist'.\" These four channels are therefore not produced by any tool at test time; they are human-annotated superpixel labels. They are also label-adjacent: milia-like cysts are a hallmark of seborrheic keratosis (non-melanocytic in ISIC-MN), and pigment network is associated with melanocytic lesions. So the TBM receives structured information that is highly predictive of the target label, while the EfficientNet baseline sees only raw pixels. This is an oracle comparison, not a comparison of tool-use frameworks. The reported ISIC-MN AUC of 91.8 versus 91.2, and the Figure 2b low-data gains, may simply reflect the free availability of ground-truth clinical cues. The main text and abstract do not disclose this; it appears only in Appendix C.2. The Camelyon17 experiment uses predicted HoVer-Net outputs and is therefore not affected, but the paper's broad \"on par or better\" claim across both modalities is not supported by the dermatology evidence as designed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the Tool Bottleneck Framework (TBF), which uses a medical VLM to select clinically relevant tools from a toolbox and then fuses the selected tools' outputs with a learned Tool Bottleneck Model (TBM) to make image-level predictions. TBF is evaluated on Camelyon17 (histopathology) and two ISIC 2017 binary tasks (dermatology), with reported accuracy/AUC on par with or better than CNNs and VLMs, plus data-efficiency and interpretability analyses. The method contribution is a learned composition mechanism for pixel-level and scalar tool outputs, enabled by a knockout-style training strategy that makes the TBM robust to arbitrary tool selections.","tokens_in":20776,"tokens_out":4071,"duration_ms":46709,"significance":"If the claims held, TBF would be a useful step toward clinically grounded and interpretable medical image classification: it generalizes Concept Bottleneck Models to spatially localized features, replaces text-based tool composition with a learned fusion network, and provides a simple strategy for handling arbitrary VLM tool selections. The authors are transparent in the appendix about the idealized nature of several dermatology tools, and the knockout proof in Appendix E is a sound adaptation of prior work. However, the headline claim—that TBF performs 'on par with or better than' black-box models and tool-use frameworks across histopathology and dermatology—is only cleanly supported on Camelyon17. Two of the three evaluation tasks rely on ground-truth dermoscopic structure annotations as 'hypothetical tool outputs' at test time, which is an oracle setup, not a realized tool-use system.","major_comments":[{"comment":"Four of the seven ISIC tools (pigment network, negative network, streaks, milia-like cysts) are not produced by learned detectors at test time; they are ground-truth dermoscopic feature annotations from the ISIC 2017 dataset. Appendix C.2 states this explicitly, calling it 'an idealized scenario in which accurate detectors ... 'exist''. These annotations are label-adjacent: milia-like cysts are strongly indicative of seborrheic keratosis (the non-melanocytic class in ISIC-MN), and pigment network is associated with melanocytic lesions. The TBM therefore receives structured label-adjacent information that the EfficientNet baseline does not. Since two of the three main tasks (ISIC-BM, ISIC-MN) and the data-efficiency experiment in Figure 2(b) depend on these oracle maps, the central claim of 'on par with or better than ... across histopathology and dermatology' is unsupported for a realize","section":"Appendix C.2, Table 1, Sections 4.1 and 5.1"},{"comment":"The data-efficiency argument is a key contribution claim, but the ISIC-MN panel is compromised by the same oracle-tool setup. The reported gains of TBF over EfficientNet in the 4–64 image regime may simply reflect that the TBM is fed ground-truth dermoscopic structure maps, whereas EfficientNet receives only raw pixels. These low-data results cannot be interpreted as evidence that the TBF framework, with realistically available tools, is more data-efficient. The claim 'TBF embodies clinically-relevant inductive biases' therefore needs to be re-established on Camelyon17 alone, or with predicted (non-oracle) ISIC tools. As written, the ISIC-MN data-efficiency result is an upper-bound experiment, not a demonstration of the framework.","section":"Figure 2(b), Section 5.2"},{"comment":"There is an internal inconsistency in the reported Camelyon17 accuracy for TBF with ImageNet pretraining: Table 1 reports 92.3%, while Table 2 reports 92.8%, and the text in D.1 says the accuracy 'increases to 92.3%'. The discrepancy needs to be resolved and the correct value reported consistently. If the difference is due to different evaluation splits or seeds, that should be stated.","section":"Appendix D.1, Table 2 vs. Table 1"},{"comment":"The paper repeatedly states that Y-Net is trained on 'the same amount and types of data' as TBF (Sections 4.3, 5.1, Appendix B). However, TBF receives five histopathology tool maps or seven dermatology tool maps, while Y-Net receives only a single segmentation mask (tumor masks for Camelyon, lesion masks for ISIC) plus raw images. This is not the same type or amount of pixel-level supervision. The comparison is still meaningful as a segmentation-for-classification baseline, but the 'same data' claim should be softened or the Y-Net training should be updated to consume the full tool-stack inputs.","section":"Appendix B, Y-Net baseline"}],"minor_comments":[{"comment":"The accuracy/AUC values are reported without confidence intervals or significance tests. Given the small differences in several comparisons (e.g., ISIC-MN 91.8 vs. 91.2), error bars or a significance statement would help the reader judge whether TBF is truly 'on par or better'.","section":"Section 5.1, Table 1"},{"comment":"The Bernoulli perturbation with parameter p = (1−α)0.5 + α s_i is described in words, but the notation is not fully formalized. Specifying the exact sampling distribution and how the placeholder value ¯z_i is applied would improve reproducibility.","section":"Section 3.3, Equation (1)"},{"comment":"The appendix uses the phrase 'hypothetical tool outputs' once, but the discussion of the four ground-truth structure maps would be clearer if the authors consistently labeled them as oracle inputs throughout the paper, including in the figure captions and table notes. Currently, Table 1 does not indicate that any ISIC results use ground-truth annotations.","section":"Appendix C.2"},{"comment":"The sentence 'TBF achieves 86.7% accuracy when trained from scratch, and increases to 92.3%' is inconsistent with Table 2, which lists 92.8% for the ImageNet-pretrained TBF. This is likely a typo, but it should be corrected.","section":"Appendix D.1"}],"recommendation":"reject","confidential_remarks":"The central problem is not that the authors hide the oracle setup—they disclose it in Appendix C.2—but that the main text and abstract advertise general claims about performance and data efficiency across histopathology and dermatology, while two of the three tasks are evaluated with ground-truth structure annotations as tool outputs. This is not a local fix: supporting the central claim would require either obtaining or training actual dermoscopic-structure detectors and re-running the ISIC experiments, or substantially narrowing the paper's claims to Camelyon17 and to an oracle upper-bound analysis. Given the title and framing, the latter would be a different paper. I would reject the manuscript in its current form, but I would encourage the authors to resubmit a version that clearly separates realized tool-use results from oracle analyses."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth your time if you care about interpretable medical classifiers. The actual new thing: instead of composing VLM-selected tools through text or code, the paper rasterizes tool outputs into spatial maps and learns a CNN fusion over them, with a knockout training trick so the fusion model accepts arbitrary tool subsets. That is a clean extension of Concept Bottleneck Models to pixel-level structured features, and it is the right instinct for medical images where spatially localized cues matter. The histopathology experiment is properly done: HoVer-Net predictions are used as tools, not ground-truth masks, and TBF beats EfficientNet 92.3 vs 88.6 on Camelyon17, with a small but consistent low-data advantage. The appendices are unusually thorough, and Appendix E adapts Nguyen et al.'s knockout result honestly.\n\nThe problem is the dermatology experiments. Four of the seven ISIC tools are not tools at inference time. For pigment network, negative network, streaks, and milia-like cysts, Appendix C.2 says they use the ground-truth ISIC superpixel annotations as hypothetical tool outputs, an idealized scenario. Those maps are label-adjacent: milia-like cysts are concentrated in seborrheic keratosis, which is the positive class in ISIC-MN. So the TBF on ISIC-MN receives structured ground-truth clinical cues that the EfficientNet baseline never sees. The headline \"on par or better\" across histopathology and dermatology therefore rests on an oracle comparison on two of three tasks. The ISIC-BM and ISIC-MN numbers, and the Figure 2b data-efficiency story, are an upper bound on tool-use value rather than a realized system. The authors disclose this only in the appendix, not in the abstract or main text, and the abstract's \"clinically-grounded predictors\" promise is not supported for dermoscopic structures without real detectors.\n\nOther soft spots are minor. Alpha and k are tuned on validation; Table 1 lacks error bars, though Figure 2 has CIs; VisProg's constant-output behavior suggests the text-composition baselines are weak rather than representative of the current tool-use class. None of that changes my read.\n\nWho is this for: people working on concept bottlenecks, VLM tool use, or interpretable medical imaging. The TBM formulation and the Camelyon17 setup are a genuine contribution, and the paper is clearly written by people who think carefully. But in current form the central claim is overbroad. A revised version that replaces the oracle derm tools with real predicted detectors, reports error bars, and limits claims to what is measured would deserve a serious look. I would send it to peer review rather than desk reject, because the methodological idea and the histopathology result merit referee time, but I would expect rejection unless the ISIC evidence is rebuilt.","headline":"The Tool Bottleneck idea is real and the Camelyon17 results are credible, but the dermatology numbers lean on ground-truth annotations passed as 'tools,' and the paper's broad claims do not survive that.","tokens_in":21272,"tokens_out":1869,"would_cite":true,"duration_ms":20980,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Composing VLM-selected clinical tools with a learned bottleneck model yields medical-image classifiers that are accurate, interpretable, and data-efficient.","keywords":["tool use","vision-language models","interpretability","medical imaging","concept bottleneck","tool composition","data efficiency","histopathology"],"falsifier":"Run the dermatology experiments with the four dermoscopic structure maps produced by detectors trained on separate data (instead of ground-truth annotations) and compare AUC; if the framework's advantage over a standard CNN vanishes or reverses, the central claim fails for dermatology.","tokens_in":20276,"feed_emoji":"🩺","tokens_out":3359,"duration_ms":30734,"temperature":0.7,"pith_summary":"The paper argues that medical image classifiers become more accurate and more interpretable when, instead of reasoning over raw pixels, a vision-language model selects clinically meaningful tools (e.g., nucleus detectors, lesion segmenters) and a small learned network fuses the tool outputs into a diagnosis. This Tool Bottleneck Framework replaces text-based tool composition, which struggles with spatially localized medical features, with a neural 'Tool Bottleneck Model.' On histopathology and dermatology tasks the framework matches or beats end-to-end CNNs and VLMs, and in data-limited regimes (4–64 labeled images) it shows the largest gains. The authors also show that the learned fusion makes tool importance measurable and decisions intervenable.","feed_headline":"Tool bottleneck model beats black-box CNNs on medical images","feed_subtitle":"A learned fusion of VLM-selected clinical tools matches or beats end-to-end models and excels with tiny datasets.","key_machinery":"The Tool Bottleneck Model (TBM) is the central object: a frozen toolbox of clinically relevant tools produces feature maps, the TBM concatenates them across channels and applies a CNN to fuse them into an image-level prediction. Tool knockout augmentation—replacing unselected tools' outputs with a constant placeholder during training—makes the TBM robust to arbitrary tool selections and simultaneously learns the full conditional and every marginal conditional of the label given tool subsets.","core_discovery":"The central claim is that composing a VLM-selected set of clinical tools through a learned bottleneck is a better inductive bias for medical image understanding than end-to-end learning or text-based tool composition. Concretely, the Tool Bottleneck Model rasterizes each tool's output into spatial maps, concatenates them, and feeds them to a CNN that outputs the prediction; tool knockout augmentation during training lets the model handle any VLM selection. Results on histopathology tumor detection and two dermatology classification tasks show accuracy/AUC on par or better than a standard CNN (e.g., 92.3% vs 88.6% on the histopathology task) and clear gains when only a few dozen labeled examp","pith_inferences":["The dermatology results rely in part on ground-truth structure annotations as hypothetical tool outputs; real deployment would require accurate detectors for pigment networks, streaks, etc., so the reported dermatology gains may reflect an upper bound.","The framework's value should extend beyond classification to other medical tasks with spatially localized features (e.g., segmentation, grading), if suitable tools exist.","A testable extension: use TBF with self-supervised or foundation-model tools to see whether the data-efficiency and interpretability benefits persist when tools are not domain-specific."],"forward_implications":["If TBF is correct, learned fusion of VLM-selected clinical tools is a viable alternative to end-to-end learning in medical imaging.","TBF's data efficiency means small clinical datasets (tens of images) can still yield usable classifiers if a relevant toolbox exists.","Tool-importance and intervention analyses give clinicians a way to inspect and correct model reasoning.","Text-based tool composition (code or natural language) is the limiting factor for medical tool-use frameworks.","Including all modality tools rather than VLM selection hurts performance, so tool selection matters."],"fun_headline_variants":["Tool bottleneck fuses clinical tools for medical image AI","Learned tool fusion beats end-to-end medical imaging","Tool Bottleneck Framework: interpretable medical image predictions","Data-hungry? Tool bottleneck works with tiny medical datasets","Tool bottleneck model rivals CNNs with fewer medical labels"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"For the dermatology tasks, four of the seven tool channels are not produced by real tools but by ground-truth dermoscopic annotations treated as hypothetical detector outputs; if such detectors are unavailable or inaccurate in practice, the reported dermatology results are not established.","fun_headline_variants_meta":{"raw":{"variants":["Tool bottleneck fuses clinical tools for medical image AI","Learned tool fusion beats end-to-end medical imaging","Tool Bottleneck Framework: interpretable medical image predictions","Data-hungry? Tool bottleneck works with tiny medical datasets","Tool bottleneck model rivals CNNs with fewer medical labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000624,"raw_usage":{"total_tokens":2778,"prompt_tokens":851,"completion_tokens":1927,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":1849}},"tokens_in":595,"tokens_out":1927,"duration_ms":13712,"temperature":1.0,"reasoning_tokens":1849,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T14:06:51.644316+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the dermatology experiments with the four dermoscopic structure maps produced by detectors trained on separate data (instead of ground-truth annotations) and compare AUC; if the framework's advantage over a standard CNN vanishes or reverses, the central claim fails for dermatology.","supporting_citations":[],"review_version":1}