pith. sign in

arxiv: 2606.23494 · v1 · pith:24UVEL2Qnew · submitted 2026-06-22 · 💻 cs.CV

Brain-Adapter: A Dual-Stream Vision-Language MIL Framework for Comprehensive 3D CT Diagnosis of Acute Intracranial Pathologies

Zhenyu Yi , Zhiyun Song , Yusong Sun , Zelin Liu , Manman Fei , Zhenhao Li , Jiaxuan Zhao , Xu Han
show 1 more author
Lichi Zhang
This is my paper

Pith reviewed 2026-06-26 08:51 UTC · model grok-4.3

classification 💻 cs.CV
keywords 3D CT diagnosisvision-language modelmultiple instance learningintracranial pathologytext-conditioned attentiondual-stream frameworkreport supervision
0
0 comments X

The pith

Brain-Adapter transfers pre-trained 2D vision-language models to 3D CT brain scans for multi-label diagnosis using only diagnostic reports and no dense annotations.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents Brain-Adapter as a dual-stream multiple instance learning framework for 3D brain CT analysis. One stream uses a text-conditioned attention mechanism driven by raw diagnostic sentences to align visual features with disease concepts from pre-trained 2D VLMs. The second stream processes the full volume under supervision from structured labels extracted by an LLM from the reports. A consistency constraint links the streams, and an uncertainty-aware module refines predictions at test time. This setup claims to deliver better performance than existing 3D models while removing the need for manual slice-level or voxel-level labels.

Core claim

The central claim is that a dual-stream MIL architecture, with text-conditioned attention using report sentences and parallel visual supervision from LLM labels, plus consistency and uncertainty refinement, successfully adapts 2D VLMs to comprehensive 3D CT diagnosis of acute intracranial pathologies without requiring dense annotations.

What carries the argument

The Text-Conditioned Attention (TCA) mechanism that treats diagnostic sentences as semantic queries to guide attention over 3D volume instances, combined with the dual-stream setup and Uncertainty-Aware Refinement (UAR) module.

If this is right

  • The method achieves higher accuracy in multi-label classification of intracranial pathologies than state-of-the-art 3D models and standard MIL approaches.
  • It enables training and deployment using only raw diagnostic reports for supervision.
  • The dual streams with consistency constraint produce more coherent predictions across visual and text-guided pathways.
  • The UAR module improves handling of ambiguous or uncertain cases during inference.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • This could reduce the cost and time of annotating large 3D medical datasets substantially.
  • Similar adapter strategies might apply to other volumetric imaging tasks like MRI or PET scans.
  • Clinical adoption could be faster if the method integrates into existing radiology workflows that already produce reports.

Load-bearing premise

That the pre-trained 2D biomedical VLMs can be transferred effectively to 3D volumes through the TCA mechanism and that LLM-extracted labels from reports are reliable enough to supervise the visual stream.

What would settle it

A controlled experiment on an independent 3D CT dataset where the proposed method shows no statistically significant improvement over a standard 3D MIL baseline trained with the same report-derived labels would falsify the performance claim.

Figures

Figures reproduced from arXiv: 2606.23494 by Jiaxuan Zhao, Lichi Zhang, Manman Fei, Xu Han, Yusong Sun, Zelin Liu, Zhenhao Li, Zhenyu Yi, Zhiyun Song.

Figure 1
Figure 1. Figure 1: Overview of Brain-Adapter. Training (Top): LoRA-adapted slice fea￾tures feed two pathways. TCA matches text embeddings with relevant slices for Open-set Alignment, while ABMIL aggregates features for Logic-set Supervision using LLM-extracted labels. A consistency loss synchronizes global representa￾tions z and h0. Inference (Bottom): UAR dynamically fuses MIL predictions with prompt-based semantic scores (… view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of the brain pathological labels. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Evaluation of zero-shot transferability and interpretability. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
read the original abstract

Automated diagnosis of 3D brain CT scans is essential for critical care, yet it remains challenging due to the heavy reliance on manual annotations and the limited semantic understanding of conventional models. While 2D foundation vision-language models (VLMs) have shown remarkable generalization, effectively transferring their representational power to 3D volumes remains an open problem. In this paper, we propose Brain-Adapter, a novel dual-stream multiple instance learning (MIL) framework that leverages pre-trained 2D biomedical VLMs and raw diagnostic reports for robust scan-level multi-label classification. Specifically, we introduce a Text-Conditioned Attention (TCA) mechanism, utilizing raw diagnostic sentences as semantic queries to dynamically align visual cues with specific disease concepts. Concurrently, a parallel visual MIL stream captures global scan characteristics, supervised by structured labels extracted via a Large Language Model (LLM). To ensure representation coherence, a consistency constraint enforces synergy between the two streams. During inference, an Uncertainty-Aware Refinement (UAR) module dynamically calibrates and fuses these dual-stream predictions to resolve ambiguous cases. Extensive experiments demonstrate that our method significantly outperforms state-of-the-art 3D models and standard MIL approaches. By eliminating the reliance on dense annotations, Brain-Adapter provides a highly scalable and clinically viable solution for 3D acute intracranial pathology analysis.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 0 minor

Summary. The manuscript introduces Brain-Adapter, a dual-stream vision-language MIL framework for 3D CT scan-level multi-label classification of acute intracranial pathologies. It transfers pre-trained 2D biomedical VLMs via a Text-Conditioned Attention (TCA) mechanism that uses raw diagnostic sentences as semantic queries, runs a parallel visual MIL stream supervised by LLM-extracted structured labels, enforces a consistency constraint between streams, and applies an Uncertainty-Aware Refinement (UAR) module at inference. The central claim is that the method significantly outperforms state-of-the-art 3D models and standard MIL approaches by removing the need for dense annotations.

Significance. If the outperformance claims hold, the work would be significant for clinical critical-care applications by enabling scalable 3D diagnosis that leverages existing 2D VLMs and report text, thereby addressing the annotation bottleneck in medical imaging AI.

major comments (1)
  1. [Abstract] Abstract: the claim that 'Extensive experiments demonstrate that our method significantly outperforms state-of-the-art 3D models and standard MIL approaches' is unsupported because the text supplies no datasets, metrics, baselines, ablation results, or quantitative evidence, which is load-bearing for the central contribution.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their review and constructive comment. We address the single major comment below.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the claim that 'Extensive experiments demonstrate that our method significantly outperforms state-of-the-art 3D models and standard MIL approaches' is unsupported because the text supplies no datasets, metrics, baselines, ablation results, or quantitative evidence, which is load-bearing for the central contribution.

    Authors: We agree that the abstract, standing alone, does not supply the supporting details (datasets, metrics, baselines, or quantitative results) needed to substantiate the performance claim. This is a fair observation. We will revise the abstract to include a concise statement of the primary dataset, key evaluation metrics, and the reported performance gains relative to the strongest baselines. The revision will be kept within standard abstract length limits while making the central claim evidence-based. revision: yes

Circularity Check

0 steps flagged

No significant circularity; architecture claims rest on external experiments

full rationale

The paper describes a dual-stream MIL architecture (TCA mechanism, LLM-supervised visual stream, consistency constraint, UAR module) but presents no equations, derivations, or parameter-fitting steps that could reduce to self-definition. Claims of outperformance are supported by experimental results on external benchmarks rather than any internal reduction to fitted inputs or self-citations. The central premise (transfer from 2D VLMs to 3D via TCA without dense annotations) is an empirical engineering claim, not a mathematical derivation chain. No load-bearing self-citation or ansatz smuggling is evident in the provided text.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review provides no explicit free parameters, axioms, or invented entities; the framework implicitly assumes transferability of 2D VLMs and reliability of LLM label extraction.

pith-pipeline@v0.9.1-grok · 5805 in / 963 out tokens · 14958 ms · 2026-06-26T08:51:15.800757+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 6 linked inside Pith

  1. [1]

    In: 2024 International Conference on Content-Based Multimedia Indexing (CBMI)

    Berrimi, M., Teoh, Y.X., Chetouani, A., Houam, L., Jennane, R.: A multi-instance learning approach for improving knee osteoarthritis diagnosis from mri data. In: 2024 International Conference on Content-Based Multimedia Indexing (CBMI). pp. 1–6. IEEE (2024)

    2024

  2. [2]

    arXiv preprint arXiv:2108.07258 (2021)

    Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M.S., Bohg, J., Bosselut, A., Brunskill, E., et al.: On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021)

    Pith/arXiv arXiv 2021

  3. [3]

    Nature medicine25(8), 1301–1309 (2019)

    Campanella, G., Hanna, M.G., Geneslaw, L., Miraflor, A., Werneck Krauss Silva, V., Busam, K.J., Brogi, E., Reuter, V.E., Klimstra, D.S., Fuchs, T.J.: Clinical- grade computational pathology using weakly supervised deep learning on whole slide images. Nature medicine25(8), 1301–1309 (2019)

    2019

  4. [4]

    The Lancet392(10162), 2388–2396 (2018)

    Chilamkurthy, S., Ghosh, R., Tanamala, S., Biviji, M., Campeau, N.G., Venugopal, V.K., Mahajan, V., Rao, P., Warier, P.: Deep learning algorithms for detection of critical findings in head ct scans: a retrospective study. The Lancet392(10162), 2388–2396 (2018)

    2018

  5. [5]

    Nature medicine24(10), 1559–1567 (2018)

    Coudray,N.,Ocampo,P.S.,Sakellaropoulos,T.,Narula,N.,Snuderl,M.,Fenyö,D., Moreira, A.L., Razavian, N., Tsirigos, A.: Classification and mutation prediction from non–small cell lung cancer histopathology images using deep learning. Nature medicine24(10), 1559–1567 (2018)

    2018

  6. [6]

    arXiv preprint arXiv:2010.11929 (2020)

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

    Pith/arXiv arXiv 2010

  7. [7]

    Eslami, S., De Melo, G., Meinel, C.: Does clip benefit visual question answering in the medical domain as much as it does in the general domain? arXiv preprint arXiv:2112.13906 (2021)

    arXiv 2021

  8. [8]

    In: International MICCAI brainlesion workshop

    Hatamizadeh, A., Nath, V., Tang, Y., Yang, D., Roth, H.R., Xu, D.: Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images. In: International MICCAI brainlesion workshop. pp. 272–284. Springer (2021)

    2021

  9. [9]

    Iclr1(2), 3 (2022)

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. Iclr1(2), 3 (2022)

    2022

  10. [10]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Hu, Q., Wei, J., Yi, Z., Yan, Z., Guo, Y., Shi, H., Ji, G.P., Li, Q., Wang, Z.: Samix: Reinforcing sam2 with semantic adapter and reference selecting policy for mix-supervised segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 17948–17958 (2026)

    2026

  11. [11]

    In: Pro- ceedings of the AAAI Conference on Artificial Intelligence

    Hu, Q., Yi, Z., Zhou, Y., Huang, F., Liu, M., Li, Q., Wang, Z.: Monobox: Tightness- free box-supervised polyp segmentation using monotonicity constraint. In: Pro- ceedings of the AAAI Conference on Artificial Intelligence. vol. 39, pp. 3572–3580 (2025)

    2025

  12. [12]

    In: International Conference on Medical Image Computing and Computer-Assisted Intervention

    Hu, Q., Yi, Z., Zhou, Y., Peng, F., Liu, M., Li, Q., Wang, Z.: Sali: Short-term align- ment and long-term interaction network for colonoscopy video polyp segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 531–541. Springer (2024) 10 Z. Yi et al

    2024

  13. [13]

    In: ICCV

    Huang, S.C., Shen, L., Lungren, M.P., Yeung, S.: Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition. In: ICCV. pp. 3942–3951 (2021)

    2021

  14. [14]

    In: International conference on machine learning

    Ilse,M.,Tomczak,J.,Welling,M.:Attention-baseddeepmultipleinstancelearning. In: International conference on machine learning. pp. 2127–2136. PMLR (2018)

    2018

  15. [15]

    In: Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Li, B., Li, Y., Eliceiri, K.W.: Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In: Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 14318–14328 (2021)

    2021

  16. [16]

    arXiv preprint arXiv:2512.12887 (2025)

    Liu, H., Georgescu, B., Zhang, Y., Yoo, Y., Baumgartner, M., Gao, R., Wang, J., Zhao, G., Gibson, E., Comaniciu, D., et al.: Revisiting 2d foundation models for scalable 3d medical image classification. arXiv preprint arXiv:2512.12887 (2025)

    Pith/arXiv arXiv 2025

  17. [17]

    The British journal of radiology89(1061), 20150849 (2016)

    Lolli, V., Pezzullo, M., Delpierre, I., Sadeghi, N.: Mdct imaging of traumatic brain injury. The British journal of radiology89(1061), 20150849 (2016)

    2016

  18. [18]

    Computer Methods And Programs In Biomedicine219, 106783 (2022)

    López-Pérez, M., Schmidt, A., Wu, Y., Molina, R., Katsaggelos, A.K.: Deep gaus- sian processes for multiple instance learning: Application to ct intracranial hem- orrhage detection. Computer Methods And Programs In Biomedicine219, 106783 (2022)

    2022

  19. [19]

    arXiv preprint arXiv:1711.05101 (2017)

    Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)

    Pith/arXiv arXiv 2017

  20. [20]

    Engineering Applications of Artificial Intelligence133, 108192 (2024)

    Neethi, A., Kannath, S.K., Kumar, A.A., Mathew, J., Rajan, J.: A comprehensive review and experimental comparison of deep learning methods for automated hem- orrhage detection. Engineering Applications of Artificial Intelligence133, 108192 (2024)

    2024

  21. [21]

    arXiv preprint arXiv:1807.03748 (2018)

    Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03748 (2018)

    Pith/arXiv arXiv 2018

  22. [22]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Rafsani, F., Shah, J., Chong, C.D., Schwedt, T.J., Wu, T.: Dinoatten3d: Slice- level attention aggregation of dinov2 for 3d brain mri anomaly classification. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3544–3553 (2025)

    2025

  23. [23]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Ridnik, T., Ben-Baruch, E., Zamir, N., Noy, A., Friedman, I., Protter, M., Zelnik- Manor, L.: Asymmetric loss for multi-label classification. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 82–91 (2021)

    2021

  24. [24]

    Handbook of clinical neurology135, 447–477 (2016)

    Rincon, S., Gupta, R., Ptak, T.: Imaging of head trauma. Handbook of clinical neurology135, 447–477 (2016)

    2016

  25. [25]

    Advances in neural information processing systems34, 2136–2147 (2021)

    Shao, Z., Bian, H., Chen, Y., Wang, Y., Zhang, J., Ji, X., et al.: Transmil: Trans- former based correlated multiple instance learning for whole slide image classifica- tion. Advances in neural information processing systems34, 2136–2147 (2021)

    2021

  26. [26]

    Academic Emergency Medicine24(1), 22–30 (2017)

    Sharp, A.L., Nagaraj, G., Rippberger, E.J., Shen, E., Swap, C.J., Silver, M.A., McCormick,T.,Vinson,D.R.,Hoffman,J.R.:Computedtomographyuseforadults with head injury: describing likely avoidable emergency department imaging based on the canadian ct head rule. Academic Emergency Medicine24(1), 22–30 (2017)

    2017

  27. [27]

    In: Proceedings of the 2022 Conference on Empir- ical Methods in Natural Language Processing

    Wang, Z., Wu, Z., Agarwal, D., Sun, J.: Medclip: Contrastive learning from un- paired medical images and text. In: Proceedings of the 2022 Conference on Empir- ical Methods in Natural Language Processing. pp. 3876–3887 (2022)

    2022

  28. [28]

    Nature638(8051), 769–778 (2025)

    Xiang, J., Wang, X., Zhang, X., Xi, Y., Eweje, F., Chen, Y., Li, Y., Bergstrom, C., Gopaulchan, M., Kim, T., et al.: A vision–language foundation model for precision oncology. Nature638(8051), 769–778 (2025)

    2025

  29. [29]

    arXiv preprint arXiv:2303.00915 (2023)

    Zhang, S., Xu, Y., Usuyama, N., Xu, H., Bagga, J., Tinn, R., Preston, S., Rao, R., Wei, M., Valluri, N., et al.: Biomedclip: a multimodal biomedical foundation Brain-Adapter 11 model pretrained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2303.00915 (2023)

    Pith/arXiv arXiv 2023

  30. [30]

    arXiv preprint arXiv:2512.14480 (2025)

    Zhao, W., Huang, Z., Feng, J., Wang, X.: Superclip: Clip with simple classification supervision. arXiv preprint arXiv:2512.14480 (2025)

    arXiv 2025

  31. [31]

    IEEE Transactions on Circuits and Systems for Video Technology34(10), 9010–9023 (2024)

    Zhao, X., Li, Z., Luo, X., Li, P., Huang, P., Zhu, J., Liu, Y., Zhu, J., Yang, M., Chang, S., et al.: Ultrasound nodule segmentation using asymmetric learning with simple clinical annotation. IEEE Transactions on Circuits and Systems for Video Technology34(10), 9010–9023 (2024)

    2024