Pith. sign in

REVIEW 4 major objections 4 minor 35 references

IP-CRR: Information Pursuit for Interpretable Classification of Chest Radiology Reports

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Pith's one-line read This paper claims that classifying chest radiology reports can be made interpretable-by-design: a short chain of fact-based questions and answers determines the prediction, with no post-hoc explanation step.

desk verdict A genuine V-IP extension to text with a transparent limitation section, but the unvalidated NLI answers are the real risk and should be tested before this can claim clinical interpretability. read the letter →

arxiv 2505.00191 v2 pith:KRXDDKFV submitted 2025-04-30 cs.CL

classification cs.CL
keywords interpretablemachinelearninginformationpursuitchestradiologyreportsnaturallanguageinferenceMIMIC-CXRtextclassificationconceptbottleneckmodelsmedicalAI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that chest-radiology-report classification can be done by an interpretable-by-design pipeline whose prediction is fully determined by a short sequence of question-answer pairs about the report. It mines 520 representative clinical facts from over 220k reports, uses a general-purpose natural language inference model to say whether each fact is present, absent, or unknown in a new report, and then uses variational Information Pursuit to ask the most informative questions before making the diagnosis. If the claim holds, the explanation is not a post-hoc approximation but the very chain of questions and answers that produced the prediction. On the MIMIC-CXR dataset, the authors report higher average precision and F1 than the black-box Flan-T5-large model and than Concept Bottleneck Models across several tasks, while remaining below a fully fine-tuned CXR-BERT on some tasks.

What carries the argument

The central mechanism is Variational Information Pursuit (V-IP): a querier network $g_\eta$ that, given the history of previous query-answer pairs, selects the query with largest conditional mutual information with the label, and a predictor network $f_\theta$ that estimates $P(Y \mid q_{1:k}(x))$ from the masked history vector. The query set itself is built by extracting over 590k facts from more than 660k sentences of MIMIC-CXR, embedding them in a 128-dimensional fact space, clustering, and keeping 520 representative, deduplicated facts. Query answers come from a fixed instruction-tuned language model prompted with the report as premise and the fact as hypothesis, producing positive, negative, or unknown; the explicit unknown answer is a new ingredient that handles reports that do not mention the fact.

What would settle it

Take a random sample of MIMIC-CXR reports, have radiologists annotate the 520 mined facts as present, absent, or unknown, and compare those labels to Flan-T5-large's NLI answers; if agreement is low, the query-answer chains are not faithful to the reports and the paper's central interpretability claim fails.

Watch

Extended reading notes

Core claim

The central claim is that a small, curated set of natural-language facts about chest radiology reports can serve as the sole medium of classification: the system never sees the report as a raw bag of tokens for prediction, but only through answers to queries about clinical findings. Given a new report, a trained querier greedily picks the next most informative fact, an NLI model answers yes, no, or unknown, and a classifier updates the disease probability from the accumulated query-answer history. The selected chain is the explanation by construction, so faithfulness is built into the model rather than approximated afterwards. On six MIMIC-CXR tasks, IP-CRR attains higher average precision and F1 than Flan-T5-large and Concept Bottleneck Models, reaches 0.95 average precision on lung opacity with about 30 queries, and needs fewer than 150 queries for most tasks at high confidence.

Load-bearing premise

The load-bearing premise is that the general-purpose NLI model, Flan-T5-large, answers each mined fact accurately enough that the selected query-answer chain reflects what the radiology report actually says; the paper states this reliance on a general-domain language model as its current limitation.

Editorial extensions

If this is right

  • The explanation shown to a clinician is exactly the list of queries and answers that drove the prediction, so auditing a decision amounts to reading a short diagnostic dialogue.
  • The model adapts how many questions it asks to the report and the confidence threshold; for example, lung opacity reaches 0.95 average precision with roughly 30 of the 520 available queries.
  • The framework does not require manually annotated query answers, since it uses automatically mined facts plus NLI, so it can be adapted to other text classification tasks where concepts are natural-language facts.
  • On the LO, CA, CM, and PN tasks, IP-CRR outperforms both an instruction-tuned black-box LLM and Concept Bottleneck Models in average precision and F1, while remaining comparable on SD and PE.
  • Because the query set and the NLI answerer are separate components, the answerer can be swapped without regenerating the mined fact set or retraining the query selection from scratch.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: if the general-purpose NLI answerer is replaced with a stronger medical-domain fact-checker, the same query set and V-IP training would likely improve both accuracy and the reliability of the explanations; the paper itself identifies reliance on the general-domain model as its current limitation.
  • My inference: since the query set is mined from MIMIC-CXR itself, the 520 selected facts may be biased toward common findings and could under-cover rare diseases; a test that measures recall on rare pathologies with an expanded query set would reveal this.
  • My inference: the explicit unknown-answer channel could be repurposed to detect settings where the report is silent on the most informative finding, which might serve as a trigger for requesting the underlying chest image or additional patient history.
  • My inference: a clinician study could compare trust and decision time when reading IP-CRR's query-answer chains versus post-hoc heatmap explanations, and because IP-CRR's explanation is faithful by construction, such a study would test whether faithfulness translates into measurable clinical benefit.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes IP-CRR, an interpretable-by-design framework for binary chest radiology report classification. The pipeline has three stages: (i) mine a query set of representative facts from a large corpus of MIMIC-CXR reports by extracting facts with a T5-small model trained on GPT-3.5/GPT-4-generated examples, embedding them with CXRFE, and applying k-means clustering with deduplication to obtain 520 queries; (ii) answer each query for each report by prompting Flan-T5-large as a natural language inference model, producing positive, negative, or unknown answers; and (iii) train a querier and a classifier under the Variational Information Pursuit (V-IP) objective, so that at test time the model sequentially selects queries and predicts the disease from the accumulated query-answer pairs. Experiments on six MIMIC-CXR classification tasks compare IP-CRR with CXR-BERT (FT-Last and FT-All), non-fine-tuned Flan-T5-large, and a Concept Bottleneck Model, and report average precision and F1 scores. The paper claims that the explanation for a prediction is, by construction, the set of selected queries and answers, and that IP-CRR outperforms black-box baselines such as Flan-T5-large and interpretable-by-design methods such as CBMs.

Significance. If the claims hold, the paper is a useful contribution to interpretable-by-design NLP for radiology: it extends V-IP from images to text, handles the practically important 'unknown' answer category, mines queries from data rather than relying on handcrafted concepts, and releases code. The analysis in Figure 3a, showing that high average precision is reached with a small fraction of the 520 queries, directly supports the efficiency of the selected explanations. The main risk is that the entire faithfulness claim rests on the correctness of the Flan-T5-large NLI answers, which are not validated anywhere; if those answers are wrong, the selected queries, the posterior, and the explanation are all wrong in a way no downstream component can correct. The empirical comparison is also weaker than the abstract suggests, because the strongest baseline (CXR-BERT FT-All) outperforms IP-CRR on most tasks and no uncertainty or significance estimates are reported.

major comments (4)
  1. [Section 3.2, 'Query Answers q(x)'] The paper's central faithfulness claim—that the explanation for a prediction is, by construction, the selected query-answer pairs (Abstract)—depends entirely on the correctness of the answers produced by Flan-T5-large, yet no validation of NLI answer quality is provided. Errors in negation ('no pneumothorax') or hedging ('cannot exclude') would propagate directly into the selected queries, the posterior, and the final explanation, and no downstream component can correct them. I request a dedicated evaluation: e.g., on a held-out sample of reports, compare Flan-T5-large's answers against an expert-annotated or rule-based reference (CheXpert labels, NegBio, or CheXbert) and report per-query accuracy, with error analysis for negation and uncertainty; additionally, an ablation replacing Flan-T5-large with a stronger (medical) NLI model would show how sensitive IP-CRR is to the answer source. Section 5's concession that this is a 'current limitation' does not remove the need for such evidence before the interpretability claim can be accepted.
  2. [Section 4.2, Table 1] The empirical claim in the Abstract that IP-CRR 'achieves higher F1 score against black-box baselines such as Flan-T5-large' is only partially supported: in Table 1, IP-CRR outperforms the non-fine-tuned Flan-T5-large and CXR-BERT (FT-Last), but the strongest black-box baseline, CXR-BERT (FT-All), achieves higher F1 on five of six tasks (all except Lung Opacity) and higher AP on five of six tasks. Moreover, no confidence intervals, standard errors, or significance tests are reported, so it is unclear whether the differences between IP-CRR, CBM, and CXR-BERT (FT-Last) are meaningful. Please report uncertainty estimates and run repeated-seed experiments, and revise the abstract and Section 4.2 to state precisely which baselines IP-CRR outperforms.
  3. [Section 3.1, 'Problem Formulation'] The framework assumes the query set Q is 'sufficient' in the sense that P(Y | x) = P(Y | {x' : q(x)=q(x'), q∈Q}), but the query set is built by k-means clustering of extracted facts with a deduplication threshold and an unspecified number of clusters, and no evidence is given that the 520 selected queries preserve this sufficiency. If the clustering discards label-relevant facts, the V-IP objective in Section 3.3 cannot in principle recover the true posterior, which weakens the guarantee that the selected query-answer chain faithfully determines the prediction. Please either verify sufficiency empirically (e.g., by measuring how the V-IP objective or classification performance saturates as the query set grows) or explicitly state that sufficiency is an assumption inherited from IP.
  4. [Section 3.2, 'Queries q'] The query set construction relies on an LLM-generated fact extraction training set (GPT-3.5/GPT-4 prompting a subset of sentences, then fine-tuning T5-small), but the paper does not report the size of that subset, the prompts used, or any quality estimate of the extracted facts. Since the 520 queries are the vocabulary of the entire interpretable pipeline, errors or biases in fact extraction will bound the achievable faithfulness regardless of downstream components. Please report these details and provide a small manual evaluation of extracted facts (e.g., precision on a random sample) or an ablation using a different fact extractor.
minor comments (4)
  1. [Section 3.3, Eq. (1)] The notation q1:k(xobs) mixes queries and answers; please define q1:k and q1:k(xobs) explicitly, e.g., as the sequence of selected queries and their answers up to step k. The term I(q(X); Y | q1:k(xobs)) conditions on observed answers, while q1:k in the V-IP objective denotes a random history; the distinction should be made precise.
  2. [Section 4.1, 'Implementation Details'] The model is trained for 1000 epochs with batch size 128, but no number of independent runs or random seeds is reported; please report the standard deviation over runs, especially because Figure 3b claims 'variance across runs'.
  3. [Figure 3b] The caption says the shaded region indicates variance across runs, but the number of runs is not given; please specify it.
  4. [Abstract] The phrase 'Chest RadiologyReport' should be 'Chest Radiology Report' (missing space).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the prediction pipeline is trained on external MIMIC-CXR labels with queries mined from unlabeled reports and answers from a pretrained NLI model.

full rationale

The paper's central claim is that predictions are made from a sequence of interpretable query-answer pairs, with the explanation being, by construction, that selected sequence. This is a design property of the model rather than a derivation that reduces to its own inputs. The disease labels come from external MIMIC-CXR annotations; the query set is mined from unlabeled reports using fact extraction and clustering; the query answers are produced by a general-domain pretrained NLI model (Flan-T5-large); and the querier and classifier are trained on query-answer-label triplets. None of these components is defined in terms of the prediction target, and no fitted parameter is renamed as a prediction. The paper does cite prior work from the same group (V-IP, references [7,8,9]) as the algorithmic backbone, but that prior work is used as a published component rather than as the sole justification for the current empirical results, which are evaluated independently on MIMIC-CXR. The acknowledged limitation that the NLI answerer is a general-domain model and may answer inaccurately is a correctness risk about the quality of the query-answer bridge, not a circularity: even an incorrect answer would leave the prediction formally a function of the query-answer chain. Thus the derivation chain is self-contained with respect to circularity concerns.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on two empirical assumptions that are not directly tested: that the 520 mined queries form a sufficient query set in the IP sense, and that Flan-T5-large's NLI answers are accurate on radiology reports. The method also introduces several hand-chosen hyperparameters (cluster count, deduplication threshold, termination rule) that affect the reported performance. No new physical or conceptual entities are introduced.

free parameters (4)
  • k-means cluster count for query set = Not directly reported; 520 representative facts after deduplication
    Determines the granularity and sufficiency of the query set; chosen by hand rather than by a criterion.
  • Deduplication cosine similarity threshold = 0.97
    Controls redundancy among the 520 selected queries; small changes could alter the query vocabulary.
  • Termination confidence threshold = 0.85
    Stops the query chain; affects the tradeoff between number of queries and accuracy in Figure 3b.
  • Maximum number of queries = 200
    Termination cap; interacts with the confidence threshold and bounds the explanation length.
assumptions (4)
  • domain assumption Sufficiency of the query set Q: P(Y | x) = P(Y | {x' in X : q(x) = q(x'), q in Q})
    Stated in Sec. 3.1 with a citation to [9]; the paper does not verify that the 520 mined facts satisfy this for the six diseases. If Q is insufficient, the classifier cannot reach the true label distribution.
  • domain assumption Flan-T5-large NLI outputs are a faithful proxy for whether a fact is entailed by the report
    Sec. 3.2 defines q(x) from the LLM's 0/1/2 output; no accuracy evaluation on radiology text is provided, and Sec. 5 admits reliance on a general-domain model.
  • domain assumption The 520 representative facts from k-means clustering on CXRFE embeddings preserve clinically relevant distinctions
    Sec. 3.2 selects the nearest fact to each cluster center and deduplicates; the choice of cluster count and embedding space determines the query vocabulary, and its clinical completeness is not audited.
  • standard math KL divergence minimization with categorical Y is empirically equivalent to cross-entropy loss
    Stated in Sec. 3.3; standard result, used to train the classifier.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IP-CRR: Information Pursuit for Interpretable Classification of Chest Radiology Reports." pith.science (2026). https://pith.science/paper/KRXDDKFV

@misc{pith2026250500191,
  author       = {Pith},
  title        = {Pith review of: IP-CRR: Information Pursuit for Interpretable Classification of Chest Radiology Reports},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KRXDDKFV}},
  note         = {Machine review of arXiv:2505.00191}
}
read the original abstract

The development of AI-based methods to analyze radiology reports could lead to significant advances in medical diagnosis, from improving diagnostic accuracy to enhancing efficiency and reducing workload. However, the lack of interpretability of AI-based methods could hinder their adoption in clinical settings. In this paper, we propose an interpretable-by-design framework for classifying chest radiology reports. First, we extract a set of representative facts from a large set of reports. Then, given a new report, we query whether a small subset of the representative facts is entailed by the report, and predict a diagnosis based on the selected subset of query-answer pairs. The explanation for a prediction is, by construction, the set of selected queries and answers. We use the Information Pursuit framework to select the most informative queries, a natural language inference model to determine if a fact is entailed by the report, and a classifier to predict the disease. Experiments on the MIMIC-CXR dataset demonstrate the effectiveness of the proposed method, highlighting its potential to enhance trust and usability in medical AI.

Figures

Figures reproduced from arXiv: 2505.00191 by the authors.

Figure 1
Figure 1. Illustration of the interpretable-by-design method. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Framework of IP-CRR, which consists of three parts: 1) query set gen [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Quantitative evaluation of IP-CRR as the number of queries increases. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Examples of query-answer chains from IP-CRR. Each row in the colored [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 29 canonical work pages

  1. [1]

    In: arXiv:2303.08774 (2024)

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., et al.: GPT-4 technical report. In: arXiv:2303.08774 (2024)

  2. [2]

    In: ACL (2004)

    Bird, S., Loper, E.: NLTK: The natural language toolkit. In: ACL (2004)

  3. [3]

    In: ECCV (2022)

    Boecking, B., Usuyama, N., Bannur, S., et al.: Making the most of text semantics to improve biomedical vision–language processing. In: ECCV (2022)

  4. [4]

    Bozkurt, S., Alkim, E., Banerjee, I., Rubin, D.L.: Automated detection of measure- ments and their descriptors in radiology reports using a hybrid natural language processing algorithm. J. Digit. Imaging (2019)

  5. [5]

    Chapman, W.W., Bridewell, W., Hanbury, P., Cooper, G.F., Buchanan, B.G.: A simple algorithm for identifying negated findings and diseases in discharge sum- maries. J. Biomed. Inform. (2001)

  6. [6]

    In: WACV (2018)

    Chattopadhay, A., Sarkar, A., Howlader, P., Balasubramanian, V.N.: Grad- CAM++: Generalized gradient-based visual explanations for deep convolutional networks. In: WACV (2018)

  7. [7]

    In: ICLR (2023)

    Chattopadhyay, A., Chan, K.H.R., Haeffele, B.D., Geman, D., Vidal, R.: Varia- tional information pursuit for interpretable predictions. In: ICLR (2023)

  8. [8]

    In: ICLR (2024) 10 Y

    Chattopadhyay, A., Chan, K.H.R., Vidal, R.: Bootstrapping variational informa- tion pursuit with large language and vision models for interpretable image classi- fication. In: ICLR (2024) 10 Y. Ge et al

Show all 35 references
  1. [9]

    IEEE TPAMI (2022)

    Chattopadhyay,A.,Slocum,S.,Haeffele,B.D.,Vidal,R.,etal.:Interpretablebyde- sign: Learning predictors by composing interpretable queries. IEEE TPAMI (2022)

  2. [10]

    In: MICCAI (2024)

    Chowdhury, T.F., Phan, V.M.H., Liao, K., et al.: AdaCBM: An adaptive concept bottleneck model for explainable and accurate diagnosis. In: MICCAI (2024)

  3. [11]

    JMLR (2024)

    Chung, H.W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., et al.: Scaling instruction-finetuned language models. JMLR (2024)

  4. [12]

    In: ICML (2023)

    Covert, I.C., Qiu, W., Lu, M., Kim, N.Y., White, N.J., Lee, S.I.: Learning to maximize mutual information for dynamic feature selection. In: ICML (2023)

  5. [13]

    Holste, G., Zhou, Y., Wang, S., Jaiswal, A., Lin, M., Zhuge, S., Yang, Y., Kim, D., Nguyen-Mau, T.H., et al.: Towards long-tailed, multi-label disease classification from chest X-ray: Overview of the CXR-LT challenge. Med. Image Anal. (2024)

  6. [14]

    In: AAAI (2019)

    Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al.: CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In: AAAI (2019)

  7. [15]

    Johnson, A.E., Pollard, T.J., Berkowitz, S.J., Greenbaum, N.R., Lungren, M.P., Deng, C.y., Mark, R.G., Horng, S.: MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Sci. Data (2019)

  8. [16]

    In: ICML (2018)

    Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., et al.: Inter- pretabilitybeyondfeatureattribution:Quantitativetestingwithconceptactivation vectors (TCAV). In: ICML (2018)

  9. [17]

    arXiv:2306.01574 (2023)

    Kim, E., Jung, D., Park, S., Kim, S., Yoon, S.: Probabilistic concept bottleneck models. arXiv:2306.01574 (2023)

  10. [18]

    In: ICML (2020)

    Koh, P.W., Nguyen, T., Tang, Y.S., Mussmann, S., Pierson, E., Kim, B., Liang, P.: Concept bottleneck models. In: ICML (2020)

  11. [19]

    In: ICML (2023)

    Li, J., Li, D., Savarese, S., Hoi, S.: BLIP-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In: ICML (2023)

  12. [20]

    arXiv:1711.05101 (2017)

    Loshchilov, I.: Decoupled weight decay regularization. arXiv:1711.05101 (2017)

  13. [21]

    NeurIPS (2017)

    Lundberg, S.M., Lee, S.I.: A unified approach to interpreting model predictions. NeurIPS (2017)

  14. [22]

    In: ACL Findings (2024)

    Messina, P., Vidal, R., Parra, D., Soto, A., Araujo, V.: Extracting and encoding: Leveraging large language models and medical knowledge to enhance radiological text representation. In: ACL Findings (2024)

  15. [23]

    In: ICLR (2023)

    Oikarinen, T., Das, S., Nguyen, L.M., Weng, T.W.: Label-free concept bottleneck models. In: ICLR (2023)

  16. [24]

    In: ICCV (2023)

    Panousis, K.P., Ienco, D., Marcos, D.: Sparse linear concept discovery models. In: ICCV (2023)

  17. [25]

    AMIA Transl

    Peng, Y., Wang, X., Lu, L., et al.: NegBio: a high-performance tool for negation and uncertainty detection in radiology reports. AMIA Transl. Sci. Proc. (2018)

  18. [26]

    Pereira, S.C., Mendonça, A.M., Campilho, A., Sousa, P., Lopes, C.T.: Automated image label extraction from radiology reports—a review. Artif. Intell. Med. (2024)

  19. [27]

    JMLR (2020)

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR (2020)

  20. [28]

    Why should I trust you?

    Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?” explaining the predictions of any classifier. In: KDD (2016)

  21. [29]

    Rudin, C.: Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. (2019)

  22. [30]

    In: ICCV (2017)

    Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., et al.: Grad-CAM: Visual explanations from deep networks via gradient-based localization. In: ICCV (2017)

  23. [31]

    In: ICML (2017) IP-CRR 11

    Shrikumar, A., Greenside, P., Kundaje, A.: Learning important features through propagating activation differences. In: ICML (2017) IP-CRR 11

  24. [32]

    In: EMNLP (2020)

    Smit, A., Jain, S., Rajpurkar, P., et al.: CheXbert: combining automatic labelers and expert annotations for accurate radiology report labeling using BERT. In: EMNLP (2020)

  25. [33]

    In: ICML (2017)

    Sundararajan, M., Taly, A., Yan, Q.: Axiomatic attribution for deep networks. In: ICML (2017)

  26. [34]

    In: ICHI (2022)

    Wang, S., Lin, M., Ding, Y., Shih, G., Lu, Z., Peng, Y.: Radiology text analysis system (RadText): architecture and evaluation. In: ICHI (2022)

  27. [35]

    In: CVPR (2023)

    Yang, Y., Panagopoulou, A., Zhou, S., et al.: Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In: CVPR (2023)

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.