REVIEW 4 major objections 4 minor 35 references
IP-CRR: Information Pursuit for Interpretable Classification of Chest Radiology Reports
T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read This paper claims that classifying chest radiology reports can be made interpretable-by-design: a short chain of fact-based questions and answers determines the prediction, with no post-hoc explanation step.
desk verdict A genuine V-IP extension to text with a transparent limitation section, but the unvalidated NLI answers are the real risk and should be tested before this can claim clinical interpretability. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is Variational Information Pursuit (V-IP): a querier network $g_\eta$ that, given the history of previous query-answer pairs, selects the query with largest conditional mutual information with the label, and a predictor network $f_\theta$ that estimates $P(Y \mid q_{1:k}(x))$ from the masked history vector. The query set itself is built by extracting over 590k facts from more than 660k sentences of MIMIC-CXR, embedding them in a 128-dimensional fact space, clustering, and keeping 520 representative, deduplicated facts. Query answers come from a fixed instruction-tuned language model prompted with the report as premise and the fact as hypothesis, producing positive, negative, or unknown; the explicit unknown answer is a new ingredient that handles reports that do not mention the fact.
What would settle it
Take a random sample of MIMIC-CXR reports, have radiologists annotate the 520 mined facts as present, absent, or unknown, and compare those labels to Flan-T5-large's NLI answers; if agreement is low, the query-answer chains are not faithful to the reports and the paper's central interpretability claim fails.
Extended reading notes
Core claim
The central claim is that a small, curated set of natural-language facts about chest radiology reports can serve as the sole medium of classification: the system never sees the report as a raw bag of tokens for prediction, but only through answers to queries about clinical findings. Given a new report, a trained querier greedily picks the next most informative fact, an NLI model answers yes, no, or unknown, and a classifier updates the disease probability from the accumulated query-answer history. The selected chain is the explanation by construction, so faithfulness is built into the model rather than approximated afterwards. On six MIMIC-CXR tasks, IP-CRR attains higher average precision and F1 than Flan-T5-large and Concept Bottleneck Models, reaches 0.95 average precision on lung opacity with about 30 queries, and needs fewer than 150 queries for most tasks at high confidence.
Load-bearing premise
The load-bearing premise is that the general-purpose NLI model, Flan-T5-large, answers each mined fact accurately enough that the selected query-answer chain reflects what the radiology report actually says; the paper states this reliance on a general-domain language model as its current limitation.
Editorial extensions
If this is right
- The explanation shown to a clinician is exactly the list of queries and answers that drove the prediction, so auditing a decision amounts to reading a short diagnostic dialogue.
- The model adapts how many questions it asks to the report and the confidence threshold; for example, lung opacity reaches 0.95 average precision with roughly 30 of the 520 available queries.
- The framework does not require manually annotated query answers, since it uses automatically mined facts plus NLI, so it can be adapted to other text classification tasks where concepts are natural-language facts.
- On the LO, CA, CM, and PN tasks, IP-CRR outperforms both an instruction-tuned black-box LLM and Concept Bottleneck Models in average precision and F1, while remaining comparable on SD and PE.
- Because the query set and the NLI answerer are separate components, the answerer can be swapped without regenerating the mined fact set or retraining the query selection from scratch.
Reading between the lines
- My inference: if the general-purpose NLI answerer is replaced with a stronger medical-domain fact-checker, the same query set and V-IP training would likely improve both accuracy and the reliability of the explanations; the paper itself identifies reliance on the general-domain model as its current limitation.
- My inference: since the query set is mined from MIMIC-CXR itself, the 520 selected facts may be biased toward common findings and could under-cover rare diseases; a test that measures recall on rare pathologies with an expanded query set would reveal this.
- My inference: the explicit unknown-answer channel could be repurposed to detect settings where the report is silent on the most informative finding, which might serve as a trigger for requesting the underlying chest image or additional patient history.
- My inference: a clinician study could compare trust and decision time when reading IP-CRR's query-answer chains versus post-hoc heatmap explanations, and because IP-CRR's explanation is faithful by construction, such a study would test whether faithfulness translates into measurable clinical benefit.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes IP-CRR, an interpretable-by-design framework for binary chest radiology report classification. The pipeline has three stages: (i) mine a query set of representative facts from a large corpus of MIMIC-CXR reports by extracting facts with a T5-small model trained on GPT-3.5/GPT-4-generated examples, embedding them with CXRFE, and applying k-means clustering with deduplication to obtain 520 queries; (ii) answer each query for each report by prompting Flan-T5-large as a natural language inference model, producing positive, negative, or unknown answers; and (iii) train a querier and a classifier under the Variational Information Pursuit (V-IP) objective, so that at test time the model sequentially selects queries and predicts the disease from the accumulated query-answer pairs. Experiments on six MIMIC-CXR classification tasks compare IP-CRR with CXR-BERT (FT-Last and FT-All), non-fine-tuned Flan-T5-large, and a Concept Bottleneck Model, and report average precision and F1 scores. The paper claims that the explanation for a prediction is, by construction, the set of selected queries and answers, and that IP-CRR outperforms black-box baselines such as Flan-T5-large and interpretable-by-design methods such as CBMs.
Significance. If the claims hold, the paper is a useful contribution to interpretable-by-design NLP for radiology: it extends V-IP from images to text, handles the practically important 'unknown' answer category, mines queries from data rather than relying on handcrafted concepts, and releases code. The analysis in Figure 3a, showing that high average precision is reached with a small fraction of the 520 queries, directly supports the efficiency of the selected explanations. The main risk is that the entire faithfulness claim rests on the correctness of the Flan-T5-large NLI answers, which are not validated anywhere; if those answers are wrong, the selected queries, the posterior, and the explanation are all wrong in a way no downstream component can correct. The empirical comparison is also weaker than the abstract suggests, because the strongest baseline (CXR-BERT FT-All) outperforms IP-CRR on most tasks and no uncertainty or significance estimates are reported.
major comments (4)
- [Section 3.2, 'Query Answers q(x)'] The paper's central faithfulness claim—that the explanation for a prediction is, by construction, the selected query-answer pairs (Abstract)—depends entirely on the correctness of the answers produced by Flan-T5-large, yet no validation of NLI answer quality is provided. Errors in negation ('no pneumothorax') or hedging ('cannot exclude') would propagate directly into the selected queries, the posterior, and the final explanation, and no downstream component can correct them. I request a dedicated evaluation: e.g., on a held-out sample of reports, compare Flan-T5-large's answers against an expert-annotated or rule-based reference (CheXpert labels, NegBio, or CheXbert) and report per-query accuracy, with error analysis for negation and uncertainty; additionally, an ablation replacing Flan-T5-large with a stronger (medical) NLI model would show how sensitive IP-CRR is to the answer source. Section 5's concession that this is a 'current limitation' does not remove the need for such evidence before the interpretability claim can be accepted.
- [Section 4.2, Table 1] The empirical claim in the Abstract that IP-CRR 'achieves higher F1 score against black-box baselines such as Flan-T5-large' is only partially supported: in Table 1, IP-CRR outperforms the non-fine-tuned Flan-T5-large and CXR-BERT (FT-Last), but the strongest black-box baseline, CXR-BERT (FT-All), achieves higher F1 on five of six tasks (all except Lung Opacity) and higher AP on five of six tasks. Moreover, no confidence intervals, standard errors, or significance tests are reported, so it is unclear whether the differences between IP-CRR, CBM, and CXR-BERT (FT-Last) are meaningful. Please report uncertainty estimates and run repeated-seed experiments, and revise the abstract and Section 4.2 to state precisely which baselines IP-CRR outperforms.
- [Section 3.1, 'Problem Formulation'] The framework assumes the query set Q is 'sufficient' in the sense that P(Y | x) = P(Y | {x' : q(x)=q(x'), q∈Q}), but the query set is built by k-means clustering of extracted facts with a deduplication threshold and an unspecified number of clusters, and no evidence is given that the 520 selected queries preserve this sufficiency. If the clustering discards label-relevant facts, the V-IP objective in Section 3.3 cannot in principle recover the true posterior, which weakens the guarantee that the selected query-answer chain faithfully determines the prediction. Please either verify sufficiency empirically (e.g., by measuring how the V-IP objective or classification performance saturates as the query set grows) or explicitly state that sufficiency is an assumption inherited from IP.
- [Section 3.2, 'Queries q'] The query set construction relies on an LLM-generated fact extraction training set (GPT-3.5/GPT-4 prompting a subset of sentences, then fine-tuning T5-small), but the paper does not report the size of that subset, the prompts used, or any quality estimate of the extracted facts. Since the 520 queries are the vocabulary of the entire interpretable pipeline, errors or biases in fact extraction will bound the achievable faithfulness regardless of downstream components. Please report these details and provide a small manual evaluation of extracted facts (e.g., precision on a random sample) or an ablation using a different fact extractor.
minor comments (4)
- [Section 3.3, Eq. (1)] The notation q1:k(xobs) mixes queries and answers; please define q1:k and q1:k(xobs) explicitly, e.g., as the sequence of selected queries and their answers up to step k. The term I(q(X); Y | q1:k(xobs)) conditions on observed answers, while q1:k in the V-IP objective denotes a random history; the distinction should be made precise.
- [Section 4.1, 'Implementation Details'] The model is trained for 1000 epochs with batch size 128, but no number of independent runs or random seeds is reported; please report the standard deviation over runs, especially because Figure 3b claims 'variance across runs'.
- [Figure 3b] The caption says the shaded region indicates variance across runs, but the number of runs is not given; please specify it.
- [Abstract] The phrase 'Chest RadiologyReport' should be 'Chest Radiology Report' (missing space).
Circularity Check
No significant circularity: the prediction pipeline is trained on external MIMIC-CXR labels with queries mined from unlabeled reports and answers from a pretrained NLI model.
full rationale
The paper's central claim is that predictions are made from a sequence of interpretable query-answer pairs, with the explanation being, by construction, that selected sequence. This is a design property of the model rather than a derivation that reduces to its own inputs. The disease labels come from external MIMIC-CXR annotations; the query set is mined from unlabeled reports using fact extraction and clustering; the query answers are produced by a general-domain pretrained NLI model (Flan-T5-large); and the querier and classifier are trained on query-answer-label triplets. None of these components is defined in terms of the prediction target, and no fitted parameter is renamed as a prediction. The paper does cite prior work from the same group (V-IP, references [7,8,9]) as the algorithmic backbone, but that prior work is used as a published component rather than as the sole justification for the current empirical results, which are evaluated independently on MIMIC-CXR. The acknowledged limitation that the NLI answerer is a general-domain model and may answer inaccurately is a correctness risk about the quality of the query-answer bridge, not a circularity: even an incorrect answer would leave the prediction formally a function of the query-answer chain. Thus the derivation chain is self-contained with respect to circularity concerns.
Assumptions & free parameters
free parameters (4)
- k-means cluster count for query set =
Not directly reported; 520 representative facts after deduplication
- Deduplication cosine similarity threshold =
0.97
- Termination confidence threshold =
0.85
- Maximum number of queries =
200
assumptions (4)
- domain assumption Sufficiency of the query set Q: P(Y | x) = P(Y | {x' in X : q(x) = q(x'), q in Q})
- domain assumption Flan-T5-large NLI outputs are a faithful proxy for whether a fact is entailed by the report
- domain assumption The 520 representative facts from k-means clustering on CXRFE embeddings preserve clinically relevant distinctions
- standard math KL divergence minimization with categorical Y is empirically equivalent to cross-entropy loss
Cite this review
Pith. "Pith review of IP-CRR: Information Pursuit for Interpretable Classification of Chest Radiology Reports." pith.science (2026). https://pith.science/paper/KRXDDKFV
@misc{pith2026250500191,
author = {Pith},
title = {Pith review of: IP-CRR: Information Pursuit for Interpretable Classification of Chest Radiology Reports},
year = {2026},
howpublished = {\url{https://pith.science/paper/KRXDDKFV}},
note = {Machine review of arXiv:2505.00191}
}
read the original abstract
The development of AI-based methods to analyze radiology reports could lead to significant advances in medical diagnosis, from improving diagnostic accuracy to enhancing efficiency and reducing workload. However, the lack of interpretability of AI-based methods could hinder their adoption in clinical settings. In this paper, we propose an interpretable-by-design framework for classifying chest radiology reports. First, we extract a set of representative facts from a large set of reports. Then, given a new report, we query whether a small subset of the representative facts is entailed by the report, and predict a diagnosis based on the selected subset of query-answer pairs. The explanation for a prediction is, by construction, the set of selected queries and answers. We use the Information Pursuit framework to select the most informative queries, a natural language inference model to determine if a fact is entailed by the report, and a classifier to predict the disease. Experiments on the MIMIC-CXR dataset demonstrate the effectiveness of the proposed method, highlighting its potential to enhance trust and usability in medical AI.
Figures
Reference graph
Works this paper leans on
-
[1]
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., et al.: GPT-4 technical report. In: arXiv:2303.08774 (2024)
arXiv 2024
-
[2]
Bird, S., Loper, E.: NLTK: The natural language toolkit. In: ACL (2004)
work page 2004
-
[3]
Boecking, B., Usuyama, N., Bannur, S., et al.: Making the most of text semantics to improve biomedical vision–language processing. In: ECCV (2022)
work page 2022
-
[4]
Bozkurt, S., Alkim, E., Banerjee, I., Rubin, D.L.: Automated detection of measure- ments and their descriptors in radiology reports using a hybrid natural language processing algorithm. J. Digit. Imaging (2019)
work page 2019
-
[5]
Chapman, W.W., Bridewell, W., Hanbury, P., Cooper, G.F., Buchanan, B.G.: A simple algorithm for identifying negated findings and diseases in discharge sum- maries. J. Biomed. Inform. (2001)
work page 2001
-
[6]
Chattopadhay, A., Sarkar, A., Howlader, P., Balasubramanian, V.N.: Grad- CAM++: Generalized gradient-based visual explanations for deep convolutional networks. In: WACV (2018)
work page 2018
-
[7]
Chattopadhyay, A., Chan, K.H.R., Haeffele, B.D., Geman, D., Vidal, R.: Varia- tional information pursuit for interpretable predictions. In: ICLR (2023)
work page 2023
-
[8]
Chattopadhyay, A., Chan, K.H.R., Vidal, R.: Bootstrapping variational informa- tion pursuit with large language and vision models for interpretable image classi- fication. In: ICLR (2024) 10 Y. Ge et al
work page 2024
Show all 35 references
-
[9]
IEEE TPAMI (2022)
Chattopadhyay,A.,Slocum,S.,Haeffele,B.D.,Vidal,R.,etal.:Interpretablebyde- sign: Learning predictors by composing interpretable queries. IEEE TPAMI (2022)
2022
-
[10]
In: MICCAI (2024)
Chowdhury, T.F., Phan, V.M.H., Liao, K., et al.: AdaCBM: An adaptive concept bottleneck model for explainable and accurate diagnosis. In: MICCAI (2024)
2024
-
[11]
JMLR (2024)
Chung, H.W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., et al.: Scaling instruction-finetuned language models. JMLR (2024)
2024
-
[12]
In: ICML (2023)
Covert, I.C., Qiu, W., Lu, M., Kim, N.Y., White, N.J., Lee, S.I.: Learning to maximize mutual information for dynamic feature selection. In: ICML (2023)
2023
-
[13]
Holste, G., Zhou, Y., Wang, S., Jaiswal, A., Lin, M., Zhuge, S., Yang, Y., Kim, D., Nguyen-Mau, T.H., et al.: Towards long-tailed, multi-label disease classification from chest X-ray: Overview of the CXR-LT challenge. Med. Image Anal. (2024)
2024
-
[14]
In: AAAI (2019)
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al.: CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In: AAAI (2019)
2019
-
[15]
Johnson, A.E., Pollard, T.J., Berkowitz, S.J., Greenbaum, N.R., Lungren, M.P., Deng, C.y., Mark, R.G., Horng, S.: MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Sci. Data (2019)
2019
-
[16]
In: ICML (2018)
Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., et al.: Inter- pretabilitybeyondfeatureattribution:Quantitativetestingwithconceptactivation vectors (TCAV). In: ICML (2018)
2018
-
[17]
arXiv:2306.01574 (2023)
Kim, E., Jung, D., Park, S., Kim, S., Yoon, S.: Probabilistic concept bottleneck models. arXiv:2306.01574 (2023)
2023 arXiv
-
[18]
In: ICML (2020)
Koh, P.W., Nguyen, T., Tang, Y.S., Mussmann, S., Pierson, E., Kim, B., Liang, P.: Concept bottleneck models. In: ICML (2020)
2020
-
[19]
In: ICML (2023)
Li, J., Li, D., Savarese, S., Hoi, S.: BLIP-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In: ICML (2023)
2023
-
[20]
arXiv:1711.05101 (2017)
Loshchilov, I.: Decoupled weight decay regularization. arXiv:1711.05101 (2017)
2017 arXiv
-
[21]
NeurIPS (2017)
Lundberg, S.M., Lee, S.I.: A unified approach to interpreting model predictions. NeurIPS (2017)
2017
-
[22]
In: ACL Findings (2024)
Messina, P., Vidal, R., Parra, D., Soto, A., Araujo, V.: Extracting and encoding: Leveraging large language models and medical knowledge to enhance radiological text representation. In: ACL Findings (2024)
2024
-
[23]
In: ICLR (2023)
Oikarinen, T., Das, S., Nguyen, L.M., Weng, T.W.: Label-free concept bottleneck models. In: ICLR (2023)
2023
-
[24]
In: ICCV (2023)
Panousis, K.P., Ienco, D., Marcos, D.: Sparse linear concept discovery models. In: ICCV (2023)
2023
-
[25]
AMIA Transl
Peng, Y., Wang, X., Lu, L., et al.: NegBio: a high-performance tool for negation and uncertainty detection in radiology reports. AMIA Transl. Sci. Proc. (2018)
2018
-
[26]
Pereira, S.C., Mendonça, A.M., Campilho, A., Sousa, P., Lopes, C.T.: Automated image label extraction from radiology reports—a review. Artif. Intell. Med. (2024)
2024
-
[27]
JMLR (2020)
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR (2020)
2020
-
[28]
Why should I trust you?
Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?” explaining the predictions of any classifier. In: KDD (2016)
2016
-
[29]
Rudin, C.: Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. (2019)
2019
-
[30]
In: ICCV (2017)
Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., et al.: Grad-CAM: Visual explanations from deep networks via gradient-based localization. In: ICCV (2017)
2017
-
[31]
In: ICML (2017) IP-CRR 11
Shrikumar, A., Greenside, P., Kundaje, A.: Learning important features through propagating activation differences. In: ICML (2017) IP-CRR 11
2017
-
[32]
In: EMNLP (2020)
Smit, A., Jain, S., Rajpurkar, P., et al.: CheXbert: combining automatic labelers and expert annotations for accurate radiology report labeling using BERT. In: EMNLP (2020)
2020
-
[33]
In: ICML (2017)
Sundararajan, M., Taly, A., Yan, Q.: Axiomatic attribution for deep networks. In: ICML (2017)
2017
-
[34]
In: ICHI (2022)
Wang, S., Lin, M., Ding, Y., Shih, G., Lu, Z., Peng, Y.: Radiology text analysis system (RadText): architecture and evaluation. In: ICHI (2022)
2022
-
[35]
In: CVPR (2023)
Yang, Y., Panagopoulou, A., Zhou, S., et al.: Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In: CVPR (2023)
2023
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.