REVIEW 3 major objections 6 minor 20 references
TMF-RSE: Tri-Modal Fusion with Regional Semantics and Evidential Uncertainty for Lung Severity Scoring
T0 review · 3 major / 6 minor · reviewed 2026-07-08 · glm-5.2
Pith's one-line read Three views beat one for lung severity scoring
desk verdict Tri-modal fusion (DINOv3 + SAM3 + LLaVA-Med) for lung severity regression shows consistent ablation gains but never validates the fusion mechanism itself against simple concatenation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
TMF-RSE architecture: DINOv3-ViT image encoder (appearance), lightweight CNN mask encoder (structure), frozen LLaVA-Med VLM encoder (semantics), tri-modal fusion with semantic gating + structural prior modulation + hierarchical fusion, evidential regression head (Normal-Inverse-Gamma distribution)
What would settle it
If the image+VLM ablation variant performed no better than image+mask, the claim that VLM-derived clinical semantics add value beyond generic feature extraction would be undermined.
Extended reading notes
Core claim
The central finding is that combining appearance, structural, and semantic features in a hierarchical fusion architecture yields measurable improvements over image-only and pairwise bi-modal approaches for continuous lung severity regression. The VLM encoder, queried with region-specific clinical prompts for three lung zones, contributes beyond what a generic image encoder provides, as shown by the gap between image+VLM and image-only ablation variants. The dual-level uncertainty design — structural prior modulation at the representation level and evidential regression at the output level — produces uncertainty scores that track prediction difficulty, confirmed by the monotonic decrease inMA
Load-bearing premise
The paper assumes that the frozen medical vision-language model, when queried with region-specific clinical prompts about lung pathology, extracts semantically meaningful clinical features that contribute to severity regression — but it only validates this through aggregate performance improvements, without examining what the VLM embeddings actually encode.
Editorial extensions
If this is right
- Clinical deployment of automated severity scoring could use the evidential uncertainty estimates to flag cases requiring radiologist review, enabling triage rather than full replacement of human assessment.
- The region-specific VLM prompting strategy (upper/middle/lower lung zones with tailored clinical queries) could generalize to other anatomical scoring tasks where pathology distribution varies by region.
- If the VLM embeddings prove to encode clinically meaningful features rather than acting as a generic image encoder, the semantic gating mechanism could be extended to provide interpretable clinical reasoning alongside severity predictions.
- The dual-level uncertainty design (representation-level + output-level) could be adopted in other medical imaging regression tasks where both segmentation quality and prediction confidence matter.
Reading between the lines
- The paper validates VLM contribution only through aggregate performance gains, not through analysis of what the VLM embeddings encode. If the VLM is acting as a generic image encoder, a lighter-weight frozen image backbone might achieve similar gains at lower computational cost.
- The three-zone lung division may not align with actual pathological distributions in diseases beyond COVID-19; finer anatomical parcellation could matter for pathologies with different spatial patterns.
- The batch-size-1 training constraint imposed by VLM processing raises questions about optimization stability and whether the reported gains would persist with larger-scale training or different optimizer configurations.
- The uncertainty sparsification analysis shows correlation between uncertainty and error but does not establish calibration — whether the uncertainty magnitudes are clinically actionable for threshold-based triage remains untested.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes TMF-RSE, a tri-modal fusion framework for lung severity scoring that combines appearance features (DINOv3-ViT), structural features (SAM3-derived masks via a lightweight CNN), and semantic features (frozen LLaVA-Med VLM with region-specific clinical prompts for upper/middle/lower lung zones). Three fusion mechanisms are introduced: semantic gating (Eq. 1), structural prior modulation (Eq. 2), and a three-stage hierarchical fusion (Sec. II-E). An evidential regression head (NIG distribution) provides aleatoric and epistemic uncertainty estimates. Experiments are conducted on the RALO chest radiograph benchmark and the Per-COVID-19 CT benchmark (validation and test phases), with modality ablations reported in Table IV.
Significance. The integration of a medical VLM as a semantic prior for continuous severity regression is a reasonable and relatively novel idea, and the dual-level uncertainty modeling (structural prior modulation at the representation level plus evidential regression at the output level) is a thoughtful design choice. The modality ablation (Table IV) shows consistent monotonic improvements across image-only, bi-modal, and full tri-modal configurations on both benchmarks, which provides evidence that each modality contributes. The uncertainty sparsification analysis (Fig. 2) is a useful sanity check for the evidential head. The work addresses a clinically relevant problem with a principled architecture.
major comments (3)
- Sec. V-B: The paper's central architectural contribution is the fusion mechanism (semantic gating, structural prior modulation, hierarchical fusion), yet no concatenation-only baseline for the same three modalities is reported. The authors explicitly defer this: 'We do not tabulate a concatenation-only baseline here; such a comparison is left for supplementary material if needed.' Table IV ablates modality subsets but not fusion strategies. Without a concatenation baseline, the reader cannot distinguish whether the performance gains over bi-modal variants come from the proposed fusion mechanisms or simply from having access to all three modalities. This is load-bearing because the fusion mechanism IS the claimed novelty. A simple late-fusion or feature-concatenation baseline using the same three encoders would directly test whether semantic gating, structural prior modulation, and the 3×
- Tables I–III: The improvements over the strongest baselines are small (RALO GE: 0.339 vs. 0.351 MAE for PViTGAtt-IP, a 3.4% relative improvement; RALO LO: 0.310 vs. 0.322, 3.7% relative). No statistical significance testing (e.g., paired t-test, bootstrap confidence intervals) is reported for any comparison. Given the modest margins and the fact that on Per-COVID-19 CT test phase (Table III), TMF-RSE does not achieve the best MAE (3.69 vs. 3.55 for two methods), significance testing is needed to support the 'outperforms' claim. The claim should also be qualified: on the test phase, TMF-RSE achieves the best PC and RMSE but not MAE.
- Sec. II-D and Table IV: The VLM encoder's contribution is validated only through aggregate performance differences (image-only vs. image+VLM). No analysis is provided of what the VLM embeddings actually encode, whether the region-specific prompts elicit distinct pathological signatures across zones, or whether the semantic gating (Eq. 1) selectively passes clinically relevant features. The VLM could be functioning as a generic image encoder. A simple diagnostic (e.g., t-SNE of VLM embeddings by zone, gate activation statistics, or comparison against a non-medical VLM) would strengthen the claim that clinical semantics are being captured.
minor comments (6)
- Sec. II-E, Stage 3: The 'synergy gate' is mentioned but not formally defined with an equation, unlike semantic gating (Eq. 1) and structural prior modulation (Eq. 2). Adding the equation would improve reproducibility.
- Sec. III-C: Batch size 1 with gradient accumulation is used, but the effective batch size is not stated. This affects reproducibility.
- Table III: The 'outperforms' language in the abstract and conclusion should be qualified, as TMF-RSE does not achieve the best MAE on the Per-COVID-19 CT test phase.
- Sec. II-D: The clinical prompts for each lung zone are described in prose but not quoted verbatim. Providing the exact prompt strings would aid reproducibility.
- Fig. 2: The y-axis range (1.0–4.0) makes it difficult to assess the sparsification behavior quantitatively. Consider reporting the MAE at specific removal fractions (e.g., 10%, 25%, 50%) in a table or adjusting the axis.
- References [8]–[11] share authorship with the submitted paper. This should be disclosed explicitly in the text or acknowledgments for transparency.
Simulated Author's Rebuttal
We thank the referee for a careful and constructive review. The referee correctly identifies that our central architectural contribution is the fusion mechanism, and we agree that a concatenation-only baseline is needed to validate it. We also accept the need for statistical significance testing and a more precise characterization of our results on the Per-COVID-19 test phase. For the VLM diagnostic analysis, we will provide what we can within our computational constraints. Below we address each major comment in turn.
read point-by-point responses
-
Referee: Sec. V-B: No concatenation-only baseline for the same three modalities is reported. The authors explicitly defer this. Without a concatenation baseline, the reader cannot distinguish whether the performance gains come from the proposed fusion mechanisms or simply from having access to all three modalities.
Authors: The referee is correct. The absence of a concatenation-only baseline using the same three encoders is a genuine gap, and our deferral to supplementary material was insufficient given that the fusion mechanism is the claimed novelty. We will add a late-fusion / feature-concatenation baseline (all three encoders, identical training protocol, simple concatenation followed by MLP regression head) to Table IV in the revised manuscript. This will directly test whether semantic gating, structural prior modulation, and hierarchical fusion provide benefits beyond mere access to all three modalities. We expect the gap to be meaningful—particularly because our semantic gating and structural prior modulation are designed to condition representations rather than simply aggregating them—but we agree the experiment is necessary to support the claim. revision: yes
-
Referee: Tables I–III: Improvements over baselines are small (3.4–3.7% relative on RALO). No statistical significance testing is reported. On Per-COVID-19 CT test phase (Table III), TMF-RSE does not achieve the best MAE (3.69 vs. 3.55). Significance testing is needed, and the 'outperforms' claim should be qualified.
Authors: We accept both points. (1) We will add paired bootstrap confidence intervals (1000 resamples) for MAE and Pearson correlation on both benchmarks, reporting p-values against the strongest baseline in each table. We will present these in a supplementary table or as annotations on the existing tables. (2) The referee is correct that on the Per-COVID-19 CT test phase (Table III), TMF-RSE achieves the best PC (0.9122) and RMSE (6.15) but not MAE (3.69 vs. 3.55 for Taiyuan_university_lab713 and PViTGAtt-IP). We will revise the manuscript text to qualify the claim accordingly, stating that TMF-RSE achieves the best PC and RMSE on the test phase but not MAE, and noting that the different metrics rank methods differently. We will avoid the unqualified word 'outperforms' where it is not supported across all metrics. revision: yes
-
Referee: Sec. II-D and Table IV: The VLM encoder's contribution is validated only through aggregate performance differences. No analysis of what the VLM embeddings actually encode, whether region-specific prompts elicit distinct pathological signatures, or whether semantic gating selectively passes clinically relevant features. A simple diagnostic (t-SNE, gate activation statistics, comparison against non-medical VLM) would strengthen the claim.
Authors: We agree that diagnostic analysis of the VLM embeddings would strengthen the paper and we will add what is feasible within our computational constraints. Specifically: (1) We will provide t-SNE visualizations of the per-zone VLM embeddings (upper, middle, lower) to show whether region-specific prompts produce separable representations. (2) We will report gate activation statistics (mean, variance, distribution) from Eq. 1, broken down by lung zone, to show whether the semantic gating is selectively modulating image features rather than acting as a uniform pass-through. (3) We will add a comparison against a non-medical VLM (standard LLaVA) as an additional ablation row, which directly tests whether the medical pretraining of LLaVA-Med contributes beyond generic vision-language features. We note one limitation: a full probing analysis of what the frozen VLM embeddings encode at a semantic concept level would require additional labeled pathological data that is not available in the Per-COVID-19 or RALO benchmarks, so our diagnostics will be representational rather than concept-level. We will be transparent about this scope. revision: partial
Circularity Check
No significant circularity; minor self-citation overlap on benchmarks and baselines, but results are independently falsifiable.
full rationale
The paper's central empirical claims are evaluated against public benchmarks (Per-COVID-19 CT, RALO) with external reference standards and independent leaderboard entries (Tables II-III). The ablation study (Table IV) tests modality contributions in a non-circular way: each variant removes a modality and measures performance change. The architectural equations (Eqs. 1-4) are standard neural network operations (semantic gating, attention modulation, evidential regression loss) that are not defined in terms of their own outputs. The uncertainty sparsification analysis (Fig. 2) is an independent validation that uncertainty tracks error, not a self-referential construction. There is notable authorship overlap: co-author Dornaika appears on the benchmark papers [5, 6] and on all four baseline methods [8-11]. However, this does not constitute circularity because: (1) the Per-COVID-19 benchmark is a public international challenge with multiple independent participants shown in Tables II-III; (2) the baselines are published methods with reproducible architectures; (3) the paper's results can be externally verified by any group with access to these public datasets. The missing concatenation baseline (acknowledged in Sec. V-B) is a completeness concern for the novelty claim, not a circularity issue. No step in the derivation chain reduces to its own inputs by construction.
Assumptions & free parameters
free parameters (7)
- lambda (evidential regularization) =
0.001
- SAM3 confidence threshold =
0.2
- Fusion module projection dimension =
512
- Cross-attention heads =
8
- Dropout rate =
0.2
- Learning rate =
1e-4
- ReduceLROnPlateau factor/patience =
0.5 / 5
assumptions (5)
- domain assumption DINOv3-ViT-S/16 pretrained on ImageNet provides useful appearance features for chest imaging without medical pretraining
- domain assumption SAM3 segmentation masks provide reliable anatomical boundaries for lung region extraction
- domain assumption LLaVA-Med embeddings encode clinically meaningful pathological semantics when queried with region-specific prompts
- domain assumption Three equal vertical zones (upper, middle, lower) adequately capture pathological heterogeneity across lung regions
- domain assumption Evidential regression (NIG distribution) provides calibrated uncertainty estimates for severity scores
Cite this review
Pith. "Pith review of TMF-RSE: Tri-Modal Fusion with Regional Semantics and Evidential Uncertainty for Lung Severity Scoring." pith.science (2026). https://pith.science/paper/RYLGGMHZ
@misc{pith2026260706356,
author = {Pith},
title = {Pith review of: TMF-RSE: Tri-Modal Fusion with Regional Semantics and Evidential Uncertainty for Lung Severity Scoring},
year = {2026},
howpublished = {\url{https://pith.science/paper/RYLGGMHZ}},
note = {Machine review of arXiv:2607.06356}
}
read the original abstract
Accurate quantification of lung disease severity from chest imaging is critical for clinical decision-making and resource allocation. We propose a tri-modal deep learning framework, TMF-RSE (Tri-Modal Fusion with Regional Semantics and Evidential Uncertainty), that combines appearance features from two-dimensional chest inputs, structural features from lung segmentation masks, and semantic features from vision-language models (VLMs) for severity quantification. Our approach employs complementary fusion mechanisms that integrate semantic guidance, structural priors, and hierarchical interactions across modalities. The model employs evidential regression to provide both severity predictions and uncertainty estimates. Experiments on the Per-COVID-19 CT and RALO datasets show that TMF-RSE outperforms recent transformer-based baselines, achieving MAE of 4.02 and Pearson correlation of 0.9629 on Per-COVID-19 validation, and 0.339 MAE / 0.973 PC on RALO geographic extent.
Figures
Reference graph
Works this paper leans on
-
[1]
A. Borghesi and R. Maroldi, “Chest x-ray severity index as a predictor of in-hospital mortality in coronavirus disease 2019: A study of 302 patients from italy,”International journal of infectious diseases, vol. 96, pp. 291–293, 2020
work page 2019
-
[2]
A. Borghesi, S. Golemi, N. Carapella, A. Zigliani, D. Farina, and R. Maroldi, “Modified brixia chest x-ray severity scoring system and correlation with intubation, non-invasive ventilation and death in a cohort of hospitalized patients with covid-19 pneumonia,”European Journal of Radiology, vol. 146, p. 110028, 2022
work page 2022
-
[3]
M. A. Warren, Z. Zhao, T. Koyama, J. A. Bastarache, C. M. Shaver, N. E. Wickersham, L. B. Ware, M. A. Matthay, and C. S. Calfee, “Severity scoring of lung oedema on the chest radiograph is associated with clinical outcomes in ards,”Thorax, vol. 73, no. 9, pp. 840–846, 2018
work page 2018
-
[4]
Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison,
J. Irvin, P. Rajpurkar, M. Ko, Y . Yu, S. Ciurea-Ilus, C. Chute, H. Mark- lund, B. Haghgoo, R. Ball, K. Shpanskayaet al., “Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison,” Proceedings of the AAAI conference on artificial intelligence, vol. 33, no. 01, pp. 590–597, 2019
work page 2019
-
[5]
Per-COVID-19: A benchmark dataset for COVID- 19 percentage estimation from CT-scans,
F. Bougourzi, C. Distante, A. Ouafi, F. Dornaika, A. Hadid, and A. Taleb-Ahmed, “Per-COVID-19: A benchmark dataset for COVID- 19 percentage estimation from CT-scans,”Journal of Imaging, vol. 7, no. 9, p. 189, 2021
work page 2021
-
[6]
F. Bougourzi, C. Distante, F. Dornaika, A. Taleb-Ahmed, A. Ouafi, A. Hadidet al., “COVID-19 infection percentage estimation from computed tomography scans: Results and insights from the international Per-COVID-19 challenge,”Sensors, vol. 24, no. 5, p. 1557, 2024
work page 2024
-
[7]
Radiographic assessment of lung opacity score dataset,
J. P. Cohen, B. Shen, A. Abbasi, M. Hoshmand-Kochi, S. Glass, H. Li, M. P. Lungren, A. S. Chaudhari, and T. Q. Duong, “Radiographic assessment of lung opacity score dataset,” 2021
work page 2021
-
[8]
B. Slika, F. Dornaika, F. Bougourzi, and K. Hammoudi, “Lung infec- tion severity prediction using transformers with conditional TransMix augmentation and cross-attention,” 2025
work page 2025
Show all 20 references
-
[9]
PViTGAtt-IP: Severity quantification of lung infections in chest x-rays and CT scans via parallel and cross-attended encoders,
——, “PViTGAtt-IP: Severity quantification of lung infections in chest x-rays and CT scans via parallel and cross-attended encoders,”IEEE Transactions on Big Data, vol. 11, no. 5, pp. 2736–2748, 2025
2025
-
[10]
Vision transformer-based model for severity quantification of lung pneumonia using chest x-ray images,
B. Slika, F. Dornaika, H. Merdji, and K. Hammoudi, “Vision transformer-based model for severity quantification of lung pneumonia using chest x-ray images,” 2023
2023
-
[11]
Multi-score prediction for lung infection severity in chest x-ray images,
B. Slika, F. Dornaika, and K. Hammoudi, “Multi-score prediction for lung infection severity in chest x-ray images,”IEEE Transactions on Emerging Topics in Computational Intelligence, vol. 9, no. 2, pp. 2052– 2058, 2025
-
[12]
Multi-modal fusion for medical image analysis: A survey,
J. Chen, Y . Li, Q. Wu, Y . Peng, and Z. Lu, “Multi-modal fusion for medical image analysis: A survey,”IEEE Transactions on Medical Imaging, vol. 40, no. 12, pp. 3456–3468, 2021
2021
-
[13]
Llava-med: Training a large language-and-vision assistant for biomedicine in one day,
C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao, “Llava-med: Training a large language-and-vision assistant for biomedicine in one day,”Advances in Neural Information Processing Systems, vol. 36, 2023
2023
-
[14]
Biomedclip: a multimodal biomedi- cal foundation model pretrained from fifteen million scientific image-text pairs,
S. Zhang, Y . Xu, N. Usuyama, J. Bagga, R. Tinn, S. Preston, R. Rao, M. Wei, N. Valluri, C. Wonget al., “Biomedclip: a multimodal biomedi- cal foundation model pretrained from fifteen million scientific image-text pairs,”arXiv preprint arXiv:2303.00915, 2023
2023 arXiv
-
[15]
Medclip: Contrastive learning from unpaired medical images and text,
Z. Wang, Z. Wu, D. Agarwal, and J. Sun, “Medclip: Contrastive learning from unpaired medical images and text,”Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3876–3887, 2022
2022
-
[16]
MedGemma technical report,
A. Sellergren, S. Kazemzadeh, T. Jaroensri, A. Kiraly, M. Traverse, T. Kohlberger, S. Xu, F. Jamil, C. Hughes, C. Lau, J. Chen, F. Mahvar et al., “MedGemma technical report,” 2025
2025
-
[17]
Deep evidential regression,
A. Amini, W. Schwarting, A. Soleimany, and D. Rus, “Deep evidential regression,”Advances in neural information processing systems, vol. 33, pp. 14 927–14 937, 2020
2020
-
[18]
A comprehensive survey on evidential deep learning and its applications,
J. Gao, M. Chen, L. Xiang, and C. Xu, “A comprehensive survey on evidential deep learning and its applications,” 2024
2024
-
[19]
Sam3: Segment anything model for medical images,
J. Chen, J. Yu, C. Ge, L. Yao, E. Xie, Y . Wu, Z. Wang, J. Kwok, P. Luo, H. Luet al., “Sam3: Segment anything model for medical images,”arXiv preprint arXiv:2401.02660, 2024
2024 arXiv
-
[20]
Siméoni, H
O. Siméoni, H. V . V o, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V . Khalidov, M. Szafraniec, S. Yi, M. Ramamonjisoa, F. Massa, D. Haz- iza, L. Wehrstedt, J. Wang, T. Darcet, T. Moutakanni, L. Sentana, C. Roberts, A. Vedaldi, J. Tolan, J. Brandt, C. Couprie, J. Mairal, H...
2025
Reviewed July 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.