{"id":"cfc300a3-d9f3-4988-be69-e4e1a5ea5ea1","arxiv_id":"2505.00191","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"IP-CRR classifies chest radiology reports by sequentially asking a small set of mined, human-readable questions and using the resulting question-answer chain as both the predictor and the explanation.","lead":"This paper introduces IP-CRR, an AI that classifies chest radiology reports by choosing a short sequence of plain-language questions, answering each one, and using those answers to predict a disease. The chosen question-answer chain is the explanation, which could make automated diagnosis easier for clinicians to check.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unvalidated NLI answers are the load-bearing link: if Flan-T5-large mis-answers queries, both prediction and explanation fail; the paper concedes this in Section 5.","rationale":"The reader's weakest assumption identifies exactly the load-bearing link. The entire interpretability story is downstream of Flan-T5-large's query answers. A bad answer source makes both the prediction and the explanation unreliable; a perfect querier and classifier cannot infer the report's content from answers that are wrong. The paper itself acknowledges this in Section 5, but treats it as a future improvement rather than a validation requirement. I considered two other candidate concerns: (1) the query set is mined from the full MIMIC-CXR dataset before the train/test split, which is a mild unsupervised leakage but unlikely to reverse the main comparison; and (2) the final classifier is a 5-layer MLP, so the query-answer chain alone does not expose the decision rule. Both are secondary: (1) would only inflate performance slightly, and (2) is partly mitigated by the sequential posterior display in Figure 4. The NLI validation issue is more fundamental because it determines whether the query-answer chain means what it appears to mean. The proposed concrete test—expert agreement on a held-out sample—directly settles it. Hence no change to the reader's conditional verdict.","tokens_in":9082,"tokens_out":6898,"duration_ms":71976,"concrete_test":"Select 100 reports from the held-out test split. For the first 20 queries selected by IP-CRR on each report, have two radiologists independently answer each query (yes/no/unknown) using the same prompt as the NLI step, and compute Cohen's kappa between Flan-T5-large answers and the radiologist consensus. If kappa < 0.6 or per-query accuracy < 80%, the query-answer chain cannot be treated as a faithful explanation, and the central interpretability claim should be revised. Additionally, for those 100 reports, recompute the downstream F1 after replacing Flan-T5-large answers with the expert-verified answers; a large drop would confirm that the reported performance is contingent on NLI reliability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that IP-CRR provides faithful, interpretable predictions rests entirely on the correctness of the query-answer pairs produced by the Flan-T5-large NLI model (Section 3.2, Figure 2 Part II). The querier and classifier in V-IP (Section 3.3) are trained on these answers, and the final 'explanation' is literally the selected query-answer chain. If the NLI model answers incorrectly—for example, by missing negation ('no pneumothorax'), misreading hedging ('possibly', 'cannot exclude'), or defaulting to 'unknown' for findings that are actually described—then the selected queries, the posterior, and the explanation are all wrong in a way that no downstream component can correct. The paper provides no validation of NLI answer quality on radiology text: no comparison against radiologist annotations, no agreement metrics, no error analysis. Section 5 explicitly concedes: 'A current limitation is the reliance on a general-domain language model for query answering.' This is not a peripheral implementation detail; it is the epistemic bridge between the report and the explanation. Without evidence that Flan-T5-large's yes/no/unknown answers align with the report's content, the abstract's claim that the explanation is 'by construction' the selected query-answer pairs is unjustified—the construction only guarantees that the prediction uses those pairs, not that the pairs are true. The concern is empirically testable and should gate acceptance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes IP-CRR, an interpretable-by-design framework for binary chest radiology report classification. The pipeline has three stages: (i) mine a query set of representative facts from a large corpus of MIMIC-CXR reports by extracting facts with a T5-small model trained on GPT-3.5/GPT-4-generated examples, embedding them with CXRFE, and applying k-means clustering with deduplication to obtain 520 queries; (ii) answer each query for each report by prompting Flan-T5-large as a natural language inference model, producing positive, negative, or unknown answers; and (iii) train a querier and a classifier under the Variational Information Pursuit (V-IP) objective, so that at test time the model sequentially selects queries and predicts the disease from the accumulated query-answer pairs. Experiments on six MIMIC-CXR classification tasks compare IP-CRR with CXR-BERT (FT-Last and FT-All), non-fine-tuned Flan-T5-large, and a Concept Bottleneck Model, and report average precision and F1 scores. The paper claims that the explanation for a prediction is, by construction, the set of selected queries and answers, and that IP-CRR outperforms black-box baselines such as Flan-T5-large and interpretable-by-design methods such as CBMs.","tokens_in":9364,"tokens_out":5498,"duration_ms":53348,"significance":"If the claims hold, the paper is a useful contribution to interpretable-by-design NLP for radiology: it extends V-IP from images to text, handles the practically important 'unknown' answer category, mines queries from data rather than relying on handcrafted concepts, and releases code. The analysis in Figure 3a, showing that high average precision is reached with a small fraction of the 520 queries, directly supports the efficiency of the selected explanations. The main risk is that the entire faithfulness claim rests on the correctness of the Flan-T5-large NLI answers, which are not validated anywhere; if those answers are wrong, the selected queries, the posterior, and the explanation are all wrong in a way no downstream component can correct. The empirical comparison is also weaker than the abstract suggests, because the strongest baseline (CXR-BERT FT-All) outperforms IP-CRR on most tasks and no uncertainty or significance estimates are reported.","major_comments":[{"comment":"The paper's central faithfulness claim—that the explanation for a prediction is, by construction, the selected query-answer pairs (Abstract)—depends entirely on the correctness of the answers produced by Flan-T5-large, yet no validation of NLI answer quality is provided. Errors in negation ('no pneumothorax') or hedging ('cannot exclude') would propagate directly into the selected queries, the posterior, and the final explanation, and no downstream component can correct them. I request a dedicated evaluation: e.g., on a held-out sample of reports, compare Flan-T5-large's answers against an expert-annotated or rule-based reference (CheXpert labels, NegBio, or CheXbert) and report per-query accuracy, with error analysis for negation and uncertainty; additionally, an ablation replacing Flan-T5-large with a stronger (medical) NLI model would show how sensitive IP-CRR is to the answer source. Section 5's concession that this is a 'current limitation' does not remove the need for such evidence before the interpretability claim can be accepted.","section":"Section 3.2, 'Query Answers q(x)'"},{"comment":"The empirical claim in the Abstract that IP-CRR 'achieves higher F1 score against black-box baselines such as Flan-T5-large' is only partially supported: in Table 1, IP-CRR outperforms the non-fine-tuned Flan-T5-large and CXR-BERT (FT-Last), but the strongest black-box baseline, CXR-BERT (FT-All), achieves higher F1 on five of six tasks (all except Lung Opacity) and higher AP on five of six tasks. Moreover, no confidence intervals, standard errors, or significance tests are reported, so it is unclear whether the differences between IP-CRR, CBM, and CXR-BERT (FT-Last) are meaningful. Please report uncertainty estimates and run repeated-seed experiments, and revise the abstract and Section 4.2 to state precisely which baselines IP-CRR outperforms.","section":"Section 4.2, Table 1"},{"comment":"The framework assumes the query set Q is 'sufficient' in the sense that P(Y | x) = P(Y | {x' : q(x)=q(x'), q∈Q}), but the query set is built by k-means clustering of extracted facts with a deduplication threshold and an unspecified number of clusters, and no evidence is given that the 520 selected queries preserve this sufficiency. If the clustering discards label-relevant facts, the V-IP objective in Section 3.3 cannot in principle recover the true posterior, which weakens the guarantee that the selected query-answer chain faithfully determines the prediction. Please either verify sufficiency empirically (e.g., by measuring how the V-IP objective or classification performance saturates as the query set grows) or explicitly state that sufficiency is an assumption inherited from IP.","section":"Section 3.1, 'Problem Formulation'"},{"comment":"The query set construction relies on an LLM-generated fact extraction training set (GPT-3.5/GPT-4 prompting a subset of sentences, then fine-tuning T5-small), but the paper does not report the size of that subset, the prompts used, or any quality estimate of the extracted facts. Since the 520 queries are the vocabulary of the entire interpretable pipeline, errors or biases in fact extraction will bound the achievable faithfulness regardless of downstream components. Please report these details and provide a small manual evaluation of extracted facts (e.g., precision on a random sample) or an ablation using a different fact extractor.","section":"Section 3.2, 'Queries q'"}],"minor_comments":[{"comment":"The notation q1:k(xobs) mixes queries and answers; please define q1:k and q1:k(xobs) explicitly, e.g., as the sequence of selected queries and their answers up to step k. The term I(q(X); Y | q1:k(xobs)) conditions on observed answers, while q1:k in the V-IP objective denotes a random history; the distinction should be made precise.","section":"Section 3.3, Eq. (1)"},{"comment":"The model is trained for 1000 epochs with batch size 128, but no number of independent runs or random seeds is reported; please report the standard deviation over runs, especially because Figure 3b claims 'variance across runs'.","section":"Section 4.1, 'Implementation Details'"},{"comment":"The caption says the shaded region indicates variance across runs, but the number of runs is not given; please specify it.","section":"Figure 3b"},{"comment":"The phrase 'Chest RadiologyReport' should be 'Chest Radiology Report' (missing space).","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The main gate for acceptance should be the NLI answer validation: as written, the faithfulness claim is only formal, not empirical. I would also ask the authors to address the partial comparison with CXR-BERT (FT-All) and to add uncertainty estimates in Table 1. The paper is within scope and the idea is promising; these are fixable with additional experiments rather than a change of approach."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: IP-CRR is a real extension of variational information pursuit to text classification, and the authors are honest about its biggest weak spot. The paper is worth a serious referee, but the referee should push for validation of the NLI answers.\n\nWhat's new and what works: The pipeline is a coherent combination of existing pieces—mining radiology facts to build the query set, using Flan-T5-large for yes/no/unknown answering, and training a V-IP querier and classifier on those answers. Applying IP to radiology reports with explicit unknown handling is new, and the query-efficiency results (e.g., 0.95 AP with 30 queries for lung opacity) are the kind of concrete evidence that matters. The authors also compare against several baselines and release code, which makes the work reproducible.\n\nSoft spots: The stress-test note gets it right: the NLI answers are the load-bearing link. If Flan-T5 systematically misses negation or hedging in radiology text, the selected queries and the posterior may be garbage, and the 'explanation' would be faithful to a broken internal reasoning process, not to the report. The authors concede this in the conclusion, which is good, but conceding a limitation is not the same as addressing it. I'd want to see some agreement check against expert annotations or a rule-based labeler (e.g., CheXpert or NegBio) before trusting the interpretability claims. Second, the quantitative comparisons lack confidence intervals and significance tests, so 'IP-CRR outperforms' is shaky for results within a few hundredths. Third, a strong medical-domain baseline like CheXbert is missing; CXR-BERT FT-All is not the only relevant comparator.\n\nProportion: These are real but fixable issues. The core method is sensible and the paper does not overclaim—it openly reports underperforming CXR-BERT FT-All on most tasks. The 'by construction' explanation claim should be read carefully: it guarantees the prediction is a function of the QA pairs, not that the answers are factually true. That nuance matters for clinical use but does not invalidate the method.\n\nWho it's for: Interpretable ML researchers, clinical NLP folks, and anyone working on chest radiology report classification. It would make a reasonable reading-group paper.\n\nRecommendation: Send it to peer review. The NLI validation should be a required revision, not a suggestion. If the authors can show the answers align with report content, this becomes a solid contribution; if not, it is a cautionary tale.","headline":"A genuine V-IP extension to text with a transparent limitation section, but the unvalidated NLI answers are the real risk and should be tested before this can claim clinical interpretability.","tokens_in":9896,"tokens_out":1489,"would_cite":true,"duration_ms":17631,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that classifying chest radiology reports can be made interpretable-by-design: a short chain of fact-based questions and answers determines the prediction, with no post-hoc explanation step.","keywords":["interpretable machine learning","information pursuit","chest radiology reports","natural language inference","MIMIC-CXR","text classification","concept bottleneck models","medical AI"],"falsifier":"Take a random sample of MIMIC-CXR reports, have radiologists annotate the 520 mined facts as present, absent, or unknown, and compare those labels to Flan-T5-large's NLI answers; if agreement is low, the query-answer chains are not faithful to the reports and the paper's central interpretability claim fails.","tokens_in":8878,"feed_emoji":"🩻","tokens_out":6062,"duration_ms":58271,"temperature":0.7,"pith_summary":"The paper tries to establish that chest-radiology-report classification can be done by an interpretable-by-design pipeline whose prediction is fully determined by a short sequence of question-answer pairs about the report. It mines 520 representative clinical facts from over 220k reports, uses a general-purpose natural language inference model to say whether each fact is present, absent, or unknown in a new report, and then uses variational Information Pursuit to ask the most informative questions before making the diagnosis. If the claim holds, the explanation is not a post-hoc approximation but the very chain of questions and answers that produced the prediction. On the MIMIC-CXR dataset, the authors report higher average precision and F1 than the black-box Flan-T5-large model and than Concept Bottleneck Models across several tasks, while remaining below a fully fine-tuned CXR-BERT on some tasks.","feed_headline":"Small query chains beat black-box models on chest reports","feed_subtitle":"The explanation is the question chain, and fewer than 150 queries often suffice for high confidence.","key_machinery":"The central mechanism is Variational Information Pursuit (V-IP): a querier network $g_\\eta$ that, given the history of previous query-answer pairs, selects the query with largest conditional mutual information with the label, and a predictor network $f_\\theta$ that estimates $P(Y \\mid q_{1:k}(x))$ from the masked history vector. The query set itself is built by extracting over 590k facts from more than 660k sentences of MIMIC-CXR, embedding them in a 128-dimensional fact space, clustering, and keeping 520 representative, deduplicated facts. Query answers come from a fixed instruction-tuned language model prompted with the report as premise and the fact as hypothesis, producing positive, negative, or unknown; the explicit unknown answer is a new ingredient that handles reports that do not mention the fact.","core_discovery":"The central claim is that a small, curated set of natural-language facts about chest radiology reports can serve as the sole medium of classification: the system never sees the report as a raw bag of tokens for prediction, but only through answers to queries about clinical findings. Given a new report, a trained querier greedily picks the next most informative fact, an NLI model answers yes, no, or unknown, and a classifier updates the disease probability from the accumulated query-answer history. The selected chain is the explanation by construction, so faithfulness is built into the model rather than approximated afterwards. On six MIMIC-CXR tasks, IP-CRR attains higher average precision and F1 than Flan-T5-large and Concept Bottleneck Models, reaches 0.95 average precision on lung opacity with about 30 queries, and needs fewer than 150 queries for most tasks at high confidence.","pith_inferences":["My inference: if the general-purpose NLI answerer is replaced with a stronger medical-domain fact-checker, the same query set and V-IP training would likely improve both accuracy and the reliability of the explanations; the paper itself identifies reliance on the general-domain model as its current limitation.","My inference: since the query set is mined from MIMIC-CXR itself, the 520 selected facts may be biased toward common findings and could under-cover rare diseases; a test that measures recall on rare pathologies with an expanded query set would reveal this.","My inference: the explicit unknown-answer channel could be repurposed to detect settings where the report is silent on the most informative finding, which might serve as a trigger for requesting the underlying chest image or additional patient history.","My inference: a clinician study could compare trust and decision time when reading IP-CRR's query-answer chains versus post-hoc heatmap explanations, and because IP-CRR's explanation is faithful by construction, such a study would test whether faithfulness translates into measurable clinical benefit."],"forward_implications":["The explanation shown to a clinician is exactly the list of queries and answers that drove the prediction, so auditing a decision amounts to reading a short diagnostic dialogue.","The model adapts how many questions it asks to the report and the confidence threshold; for example, lung opacity reaches 0.95 average precision with roughly 30 of the 520 available queries.","The framework does not require manually annotated query answers, since it uses automatically mined facts plus NLI, so it can be adapted to other text classification tasks where concepts are natural-language facts.","On the LO, CA, CM, and PN tasks, IP-CRR outperforms both an instruction-tuned black-box LLM and Concept Bottleneck Models in average precision and F1, while remaining comparable on SD and PE.","Because the query set and the NLI answerer are separate components, the answerer can be swapped without regenerating the mined fact set or retraining the query selection from scratch."],"supporting_citations":[{"why":"Defines Information Pursuit and the sufficiency condition that justifies restricting predictions to query answers.","marker":"[9]"},{"why":"Supplies the V-IP training objective and the learned querier and classifier networks that carry out sequential query selection.","marker":"[7]"},{"why":"Shows how to pair IP with large-language-model-based query answering, the template IP-CRR adapts from images to text with an unknown-answer channel.","marker":"[8]"},{"why":"Provides the fact-extraction model and fact encoder CXRFE used to mine and cluster representative queries from radiology reports.","marker":"[22]"},{"why":"Serves as the NLI model that answers each query and also as the black-box Flan-T5-large baseline in the comparison.","marker":"[11]"},{"why":"Provides the MIMIC-CXR dataset whose reports are mined for facts and whose studies are used for evaluation.","marker":"[15]"},{"why":"Defines Concept Bottleneck Models, the interpretable-by-design baseline that IP-CRR is compared against.","marker":"[18]"},{"why":"Supplies the multi-label ground-truth disease annotations used for training and evaluation.","marker":"[13]"},{"why":"Provides the CXR-BERT black-box baseline, whose fully fine-tuned variant is the strongest comparator in the results.","marker":"[3]"}],"fun_headline_variants":["Ask 150 facts, get chest diagnosis + explanation","Chest reports explained: queries replace black boxes","30 queries match black-box lung opacity accuracy","Interpretable chest AI: facts, not feature black boxes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the general-purpose NLI model, Flan-T5-large, answers each mined fact accurately enough that the selected query-answer chain reflects what the radiology report actually says; the paper states this reliance on a general-domain language model as its current limitation.","fun_headline_variants_meta":{"raw":{"variants":["Ask 150 facts, get chest diagnosis + explanation","Chest reports explained: queries replace black boxes","30 queries match black-box lung opacity accuracy","Interpretable chest AI: facts, not feature black boxes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000153,"raw_usage":{"total_tokens":1183,"prompt_tokens":899,"completion_tokens":284,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":223}},"tokens_in":515,"tokens_out":284,"duration_ms":3361,"temperature":1.0,"reasoning_tokens":223,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:49:03.337419+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of MIMIC-CXR reports, have radiologists annotate the 520 mined facts as present, absent, or unknown, and compare those labels to Flan-T5-large's NLI answers; if agreement is low, the query-answer chains are not faithful to the reports and the paper's central interpretability claim fails.","supporting_citations":[{"cited_title":"IEEE TPAMI (2022)","cited_arxiv_id":null,"evidence_quote":"Defines Information Pursuit and the sufficiency condition that justifies restricting predictions to query answers."},{"cited_title":"In: ICLR (2023)","cited_arxiv_id":null,"evidence_quote":"Supplies the V-IP training objective and the learned querier and classifier networks that carry out sequential query selection."},{"cited_title":"In: ICLR (2024) 10 Y","cited_arxiv_id":null,"evidence_quote":"Shows how to pair IP with large-language-model-based query answering, the template IP-CRR adapts from images to text with an unknown-answer channel."},{"cited_title":"In: ACL Findings (2024)","cited_arxiv_id":null,"evidence_quote":"Provides the fact-extraction model and fact encoder CXRFE used to mine and cluster representative queries from radiology reports."},{"cited_title":"JMLR (2024)","cited_arxiv_id":null,"evidence_quote":"Serves as the NLI model that answers each query and also as the black-box Flan-T5-large baseline in the comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MIMIC-CXR dataset whose reports are mined for facts and whose studies are used for evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the multi-label ground-truth disease annotations used for training and evaluation."},{"cited_title":"In: ECCV (2022)","cited_arxiv_id":null,"evidence_quote":"Provides the CXR-BERT black-box baseline, whose fully fine-tuned variant is the strongest comparator in the results."}],"review_version":1}