{"id":"aed963a2-5b0f-4952-b05c-6c28bb849eab","arxiv_id":"2602.13430","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A CXR-LT 2026 challenge solution combining reweighted training, class-aware sampling, ensembling, and CLIP prompt scoring ranked first on the development leaderboard for long-tailed and zero-shot chest X-ray classification.","lead":"This paper describes the first-place CXR-LT 2026 development-phase solution for long-tailed chest X-ray classification and zero-shot recognition of unseen diseases. It reports macro-mAPs of 0.583 and 0.467 on the two tasks, but the hidden test set remains unreleased.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Task 2 zero-shot claim is not established: WhyXrayCLIP's MIMIC-CXR pre-training likely contains the six OOD findings.","rationale":"The reader's weakest assumption was about unverified transfer of WhyXrayCLIP's embeddings to the six OOD concepts. I sharpen this into a concrete, more severe concern: the model's pre-training data itself may contain those concepts, making the zero-shot claim invalid. This is the single most load-bearing issue because the Task 2 result (0.467 mAP, rank 1) is the paper's headline zero-shot contribution; if the pre-training corpus includes the OOD terms, the result demonstrates memorization or weak supervision, not compositional zero-shot generalization. The concern is directly testable by counting OOD terms in MIMIC-CXR reports. I found no other internal inconsistency that threatens the Task 1 leaderboard claim: the method description is coherent, and the tables support the reported ranks. The paper honestly limits claims to the development phase and notes poor calibration. The CONDITIONAL verdict remains appropriate, but the condition should now be the leakage check; if the test confirms leakage, the central zero-shot claim is undermined and the paper's conclusion would need revision.","tokens_in":5740,"tokens_out":7359,"duration_ms":70177,"concrete_test":"Download the MIMIC-CXR image-report pairs used to fine-tune WhyXrayCLIP (or the released training set of the model) and count exact token occurrences of 'scoliosis', 'osteopenia', 'bulla', 'infarction', 'adenopathy', and 'goiter' in the report text. If any term appears in more than 10 unique reports, the zero-shot premise is violated. As a confirmatory control, rerun the Task 2 evaluation with the base OpenCLIP ViT-L/14 (without CXR fine-tuning) and the same prompts; if the fine-tuned model's mAP is substantially higher, the difference is attributable to pre-training on OOD-related text.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The load-bearing assumption is in §3.2: WhyXrayCLIP is 'further fine-tuned on large-scale chest X-Ray image-report pairs from MIMIC-CXR' and then used to score six OOD findings (Scoliosis, Osteopenia, Bulla, Infarction, Adenopathy, Goiter) via cosine similarity to text prompts. The paper asserts this is zero-shot because no OOD labels or examples appear in the challenge training split. But the pre-training corpus is MIMIC-CXR radiology reports, which are free text describing exactly these findings. Terms like 'scoliosis', 'adenopathy', 'bulla', and 'goiter' are standard CXR report vocabulary and appear frequently in MIMIC-CXR. Therefore the VLM has likely been trained on image–report pairs that explicitly mention these classes, meaning the model has seen both images and textual descriptions of the OOD concepts. The large Task 2 margin (0.467 vs 0.365 runner-up) may reflect this pre-training leakage rather than zero-shot transfer. The paper provides no check that the six terms are absent from the pre-training data, so the central claim of zero-shot OOD recognition is not established. If leakage is present, the mAP numbers are not a valid measure of zero-shot generalization.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes the authors' submission to the CXR-LT 2026 challenge, which consists of two tasks on a PadChest-based benchmark: Task 1 is long-tailed multi-label classification over 30 in-distribution findings, and Task 2 is zero-shot recognition of 6 out-of-distribution findings. For Task 1, the proposed pipeline combines a Distribution-Balanced loss, class-aware sampling, two ConvNeXtV2-Base models with different heads, test-time augmentation, weighted ensembling, and a normal-gating post-processing step. For Task 2, the authors use WhyXrayCLIP, a CXR-specialized vision-language model, and score each OOD finding by cosine similarity to hand-written text prompts, followed by a scaled sigmoid. The paper reports rank 1 on the public development leaderboard for both tasks, with macro mAP 0.583 (Task 1) and 0.467 (Task 2), and concludes that the proposed methods handle supervision scarcity effectively. The results are limited to the public development set because the hidden test set was not released at submission time.","tokens_in":6088,"tokens_out":5513,"duration_ms":53587,"significance":"If the leaderboard numbers are taken at face value, this is a strong challenge submission: the reported Task 1 margin over the runner-up is 0.048 mAP and the Task 2 margin is 0.102 mAP, which is substantial for this benchmark. The paper is clearly organized and provides a useful recipe for combining imbalance-aware training, ensembling, and CLIP-based zero-shot scoring. The code and model availability are also positive. However, the scientific contribution as a standalone paper is currently limited. There is no ablation study, no non-leaderboard baseline, no error bar, and no per-class/tail breakdown, so the internal causes of the reported gains are not established. More importantly, the zero-shot claim in Task 2 rests on the assumption that the six OOD concepts are absent from WhyXrayCLIP's pre-training corpus, but the paper does not verify this; since WhyXrayCLIP is fine-tuned on MIMIC-CXR image-report pairs, this assumption is questionable. The paper's value is therefore contingent on additional analysis and experiments.","major_comments":[{"comment":"The zero-shot OOD claim is not established. WhyXrayCLIP is described as 'fine-tuned on large-scale chest X-Ray image-report pairs from MIMIC-CXR.' The six OOD findings (Scoliosis, Osteopenia, Bulla, Infarction, Adenopathy, Goiter) are standard radiology terms that plausibly appear frequently in MIMIC-CXR reports. The paper only establishes that these labels are absent from the challenge training split, not that the concepts are absent from the representation-learning corpus. If the model has seen image-report pairs mentioning these findings, the reported Task 2 mAP of 0.467 measures retrieval of seen concepts, not zero-shot transfer. The authors should quantify the occurrence of these terms in the pre-training data, demonstrate absence, or provide a controlled experiment (e.g., an embedding model whose pre-training corpus excludes these concepts). Without this, the central Task 2 claim i","section":"§3.2, §4.2, Conclusion"},{"comment":"The paper's central claim is an empirical ranking, but no ablation study is reported. It is impossible to attribute the Task 1 mAP of 0.583 to any of the proposed components: Distribution-Balanced loss, class-aware sampling, CSRA head, TTA, ensemble weights, or normal gating. Each could be contributing positively, negatively, or negligibly. Similarly, for Task 2, there is no comparison against simpler alternatives such as standard CLIP, a frozen CXR encoder, or prompt-only baselines without ensembling. The authors should provide component ablations on the public development set, ideally with multiple runs or at least a clear statement of variance, to support the causal claims made in the abstract and conclusion.","section":"§4, Tables 1 and 2"},{"comment":"The stated goal of Task 1 is 'improving recognition of tail classes while maintaining stable performance on frequent findings,' but no per-class or class-frequency-stratified results are shown. Macro mAP alone does not reveal whether the improvement over the runner-up comes from tail classes, head classes, or an overall shift. The absence of this breakdown is particularly important because the proposed methods are explicitly designed for tail-class robustness. Please report per-class AP or at least head/mid/tail group AP for the proposed method and for a baseline model without the imbalance-aware components.","section":"§4.1, Task 1"}],"minor_comments":[{"comment":"The scaled sigmoid with α=5 is a monotone transformation of the cosine similarity, so it cannot change macro-mAP, which depends only on ranking. If α is intended to improve calibration or thresholded metrics, the paper should state this explicitly; as written, the method description implies it is part of the mAP-winning recipe.","section":"§3.2"},{"comment":"The reported mECE of 0.928 is extremely high, and the paper only calls it 'suboptimal.' Please clarify how mECE is computed and, if possible, show a reliability diagram or discuss why the predictions are so poorly calibrated. This is not required for the leaderboard claim but is important for the clinical motivation.","section":"§4.1"},{"comment":"The exact prompt list for the six OOD categories is not included in the paper; the authors refer to the code repository. Since zero-shot performance is highly sensitive to prompt wording, the prompts should be fully listed in the paper or a supplementary appendix for reproducibility.","section":"§3.2"},{"comment":"The CSRA head is mentioned but not described; please provide a brief formulation or a precise reference to the equations in [9] so the architecture is self-contained.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is best read as a challenge report. The leaderboard ranking is a legitimate result, but the scientific claims go beyond the evidence presented. The zero-shot leakage concern is the most serious issue and is testable: the authors should be able to compute term frequencies in MIMIC-CXR or construct a control model. The lack of ablations further weakens the internal validity. I do not see a circularity problem in the mathematical sense, but the paper's strongest claims are currently supported only by leaderboard tables."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a capable challenge report, and the rank-1 leaderboard numbers are real. But the Task 2 claim — zero-shot OOD recognition — does not hold up as stated. The six OOD concepts are common CXR findings, and WhyXrayCLIP is fine-tuned on MIMIC-CXR image-text pairs, which almost certainly contain those exact terms. That is leakage, not zero-shot transfer, and the paper gives no check that the concepts are absent from pre-training.\n\nWhat the paper does well: the pipeline is clearly specified and reproducible from the text. DB loss, repeat-factor sampling, TTA, ensembling, normal gating, and CLIP prompt ensembling are all established tools, and the specific configuration is a legitimate engineering contribution for the CXR-LT benchmark. The tables support the rank-1 claim on the public development split. The authors also state the key limitation: the hidden test is unreleased, and calibration is poor. That honesty helps.\n\nSoft spots: no ablations, no error bars, no per-class breakdown. The claimed tail-class improvement from Task 1 is asserted, not shown. The Task 2 weakness is more serious. Pre-training on MIMIC-CXR means the model has seen images and text describing scoliosis, adenopathy, bulla, goiter, etc. The 0.102 margin over the runner-up may simply reflect memorized concept alignment. Without a check of the pre-training corpus, or a control experiment that removes those concepts, the zero-shot framing is not established. Also, many hyperparameters (alpha, beta, T, rmax, alpha_ng, prompt wording) were tuned on the public leaderboard; with unreleased test, overfitting is a live concern.\n\nWho is this for? Anyone competing in CXR-LT or building a long-tailed CXR classifier would get a useful recipe. As a scientific claim about zero-shot generalization, it falls short. I would not cite it as evidence of zero-shot ability, but I would cite it as a baseline if I entered the challenge. It deserves referee time, mostly so a referee can force the leakage check. If the authors can show the OOD terms are absent or rare in pre-training, the result becomes meaningful. Without that, treat Task 2 numbers with suspicion.","headline":"A solid leaderboard recipe for CXR-LT 2026, but the Task 2 zero-shot claim is not established: the backbone was fine-tuned on MIMIC-CXR, which almost certainly contains the OOD concepts.","tokens_in":6580,"tokens_out":2167,"would_cite":false,"duration_ms":21468,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that combining class-rebalanced training with CLIP-style text-prompt scoring fixes both rare-class imbalance and zero-shot recognition of unseen chest X-ray findings, and reports first place on both CXR-LT 2026 tasks.","keywords":["chest X-ray","long-tailed learning","multi-label classification","zero-shot learning","out-of-distribution detection","Distribution-Balanced loss","CLIP","CXR-LT 2026"],"falsifier":"Take the public development images, compute the six OOD prompt-similarity scores, and compare against ground-truth labels on an independent dataset that contains those six findings; if the AUC is near chance, the zero-shot claim collapses. Also, rerun the identical Task 1 pipeline on the hidden test set and check whether the macro-mAP gap over the runner-up holds.","tokens_in":5646,"feed_emoji":"🩻","tokens_out":4508,"duration_ms":39522,"temperature":0.7,"pith_summary":"The paper addresses two supervision-scarce chest X-ray tasks from the CXR-LT 2026 challenge: classifying 30 in-distribution findings under extreme label imbalance, and detecting six out-of-distribution findings with no training examples at all. For the first task it combines a Distribution-Balanced loss, class-aware sampling, test-time augmentation, ensembling, and a 'normal gating' post-processing step. For the second it uses a chest-X-ray-tuned CLIP model to match images against generic text prompts for each unseen disease. The authors report that this solution ranks first on the public development leaderboard on both tasks, with macro mAP of 0.583 and 0.467. A sympathetic reader would care because these are realistic clinical conditions—rare diseases and never-seen findings—where standard supervised models fail.","feed_headline":"Chest X-ray pipeline ranks first on long-tail and zero-shot challenges","feed_subtitle":"Balanced loss plus CLIP text prompts handle rare and never-seen findings without extra labels.","key_machinery":"Two mechanisms carry the argument. (1) The Distribution-Balanced loss reweights classes by effective number of samples and adds a positive-label margin, paired with a repeat-factor sampler that oversamples images containing rare positives; inference adds TTA, weighted ensembling of two ConvNeXtV2-Base heads, and normal gating that suppresses abnormal scores when the Normal probability is high. (2) For zero-shot OOD recognition, the paper treats class names as text prompts, encodes them with WhyXrayCLIP (a CLIP-style model that learns a shared embedding space for images and text, fine-tuned on chest X-ray reports), averages image-text cosine similarities over prompt variations, and maps the r","core_discovery":"The central claim is that a single combined pipeline can handle both supervision regimes without OOD labels: Distribution-Balanced loss and class-aware sampling improve tail-class recognition, while test-time augmentation, ensembling, and normal gating clean up predictions; separately, a CXR-specialized CLIP model yields zero-shot OOD probabilities by averaging cosine similarity to prompt embeddings and sharpening through a scaled sigmoid. On the CXR-LT 2026 public development set, the authors report macro mAP of 0.583 (Task 1) and 0.467 (Task 2), both first on the leaderboard.","pith_inferences":["The hidden test set is the real check; a large development-set gap can shrink when the label distribution shifts, so the 0.583/0.467 numbers should be read as development-phase evidence rather than verified generalization.","The same prompt-ensembling mechanism might transfer to other radiology domains (CT, mammography) provided a domain-aligned vision-language model exists.","Because normal gating uses a single hand-set exponent (0.5), replacing it with a learned prior could further improve calibration, which the paper itself flags as weak.","The zero-shot Task 2 approach is essentially a nearest-prototype classifier in embedding space; if the six OOD concepts cluster poorly in the embedding space, the method would fail silently, making embedding-space analysis a cheap diagnostic."],"forward_implications":["If correct, the same training recipe can be reused for any long-tailed multi-label CXR dataset: reweight by effective numbers, oversample tail positives, and gate abnormal scores by the Normal-class probability.","If correct, zero-shot detection of unseen findings can be operationalized with fixed text prompts and a CXR-tuned vision-language model, eliminating the need for annotated OOD data.","If correct, the development-leaderboard scores imply that both tasks are solvable to a degree that standard supervised and unsupervised baselines do not reach.","The paper's own reported high calibration error (mECE) on Task 1 is a direct corollary that the method sacrifices calibration for ranking performance."],"fun_headline_variants":["Imbalance-aware CXR model tops long-tail and zero-shot tasks","Single CXR pipeline wins rare-disease and unseen-finding benchmarks","Tail-class loss plus CLIP prompts lead CXR challenge","CXR-LT: first place with long-tail and zero-shot without OOD labels","Balanced loss and CLIP embeddings beat CXR supervision scarcity"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"Task 2's scores rely on the assumption that WhyXrayCLIP's text embeddings for the six OOD disease names already capture their visual appearance on chest X-rays, so cosine similarity to those prompts is a valid proxy for disease presence without any OOD training examples.","fun_headline_variants_meta":{"raw":{"variants":["Imbalance-aware CXR model tops long-tail and zero-shot tasks","Single CXR pipeline wins rare-disease and unseen-finding benchmarks","Tail-class loss plus CLIP prompts lead CXR challenge","CXR-LT: first place with long-tail and zero-shot without OOD labels","Balanced loss and CLIP embeddings beat CXR supervision scarcity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000567,"raw_usage":{"total_tokens":2516,"prompt_tokens":731,"completion_tokens":1785,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":475,"completion_tokens_details":{"reasoning_tokens":1707}},"tokens_in":475,"tokens_out":1785,"duration_ms":11095,"temperature":1.0,"reasoning_tokens":1707,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T23:31:02.070337+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the public development images, compute the six OOD prompt-similarity scores, and compare against ground-truth labels on an independent dataset that contains those six findings; if the AUC is near chance, the zero-shot claim collapses. Also, rerun the identical Task 1 pipeline on the hidden test set and check whether the macro-mAP gap over the runner-up holds.","supporting_citations":[],"review_version":1}