Pith. sign in

REVIEW 2 major objections 6 minor 74 references

Unlocking the Potential of Weakly Labeled Data: A Co-Evolutionary Learning Framework for Abnormality Detection and Report Generation

T0 review · 2 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A detector and a report generator trained in a co-evolutionary loop improve each other on chest X-rays, achieving state-of-the-art detection and report quality from mostly report-only data.

desk verdict A well-ablated CXR detection and report generation co-evolution framework with real but modest gains, undercut by a concrete negative-supervision feedback path that can erase hard positives and by missing robustness statistics. read the letter →

arxiv 2412.13599 v1 pith:T5IBZUV3 submitted 2024-12-18 cs.CV cs.CL

Jinghan Sun , Dong Wei , Zhe Xu , Donghuan Lu , Hong Liu , Hong Wang , Sotirios A. Tsaftaris , Steven McDonagh
show 2 more authors
Yefeng Zheng Liansheng Wang
This is my paper · ORCID
classification cs.CVcs.CL
keywords chestX-rayabnormalitydetectionradiologyreportgenerationsemi-supervisedlearningco-evolutionarytrainingteacher-studentdistillationpseudo-labelrefinementvision-language
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that two clinical AI tasks on chest X-rays—localizing abnormalities and writing the radiology report—can be trained together so each task makes the other better. It proposes a co-evolutionary loop in which a detector and a report generator alternately train, with the generator filtering and enriching the detector's pseudo-labels on weakly labeled images and the detector feeding its detected boxes and categories into the generator. Using only about 1,000 box-annotated images alongside more than 100,000 report-only images from MIMIC-CXR, the paper reports state-of-the-art detection accuracy and improved report quality on two benchmarks. The practical stake is that large archives of unannotated X-rays with text reports could be exploited to train fine-grained localizers, reducing the burden of manual bounding-box annotation.

What carries the argument

The load-bearing mechanism is the co-evolutionary training loop built on teacher-student distillation. Three components carry the argument: SA-NMS (self-adaptive non-maximum suppression) runs NMS on the union of teacher pseudo-boxes and student predictions so confident student boxes replace stale teacher ones; GIP (generator-guided information propagation) concatenates generator features to detector features and keeps a detector pseudo-label only if its category appears in the generator's class-token prediction; DIP (detector-guided information propagation) prepends detected abnormality tokens and location embeddings to the generator's input and adds an auxiliary multi-label classification supervised by detector pseudo-labels. The alternation of GIP and DIP across iterations is what converts weakly labeled report-only data into detection supervision and box annotations into report supervision.

What would settle it

Measure per-category recall on small or heavily overlapping lesions across co-evolution iterations, for example cardiomegaly overlapped by consolidation as in the paper's Fig. 9(a)–9(b): if enabling the generator-guided filter lowers recall on such categories compared with a detector trained without GIP, the filtering is sacrificing true positives the generator also misses, and the mutual-promotion claim for those lesion types would be refuted.

Watch

Extended reading notes

Core claim

On the paper's own terms, CoE-DG demonstrates that abnormality detection and report generation are mutually informative rather than independent: a detector trained with report-guided pseudo-label filtering reaches mAP@0.5 of 37.89 on MS-CXR and 20.19 on PD-CXR, exceeding the best prior semi-supervised detector; the same detector's outputs, fed as abnormality tokens and location embeddings, lift BLEU-4 and clinical AUC of three different report generators over their baselines. The improvement is attributed to a closed loop where each model's weaknesses are corrected by the other's strengths—GIP removes detector pseudo-labels whose categories the generator does not confirm, and DIP supplies the generator with localized evidence the whole-image encoder tends to miss.

Load-bearing premise

The loop assumes the two models' pseudo-labels on unannotated images are jointly reliable, so when both models share a blind spot—tiny lesions, overlapping findings—each round of co-evolution can reinforce the same omission instead of correcting it.

Editorial extensions

If this is right

  • If the central claim holds, institutions with large archives of CXR images and free-text reports can train fine-grained abnormality localizers without manually drawing boxes on all images.
  • The same detector's outputs can simultaneously improve report generation, so a single inference pipeline can output both bounding boxes and a full report at negligible extra compute.
  • The framework's generality claim is supported by experiments with three different report generator backbones (R2Gen, R2GenCMN, ORGan), each improved by DIP.
  • Because the authors show the biggest gains at 25% and 50% of the fully labeled data, the method is most valuable in low-annotation regimes.
  • The reported gains over Soft Teacher (up to 2.07% mAP) indicate the mechanism extracts signal from reports that pure image-based semi-supervised learning cannot.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The filtering rule in GIP (keep only detector pseudo-labels whose category the generator also predicts) is conservative: it likely removes true positives that both models miss, so extending it to a soft weighting scheme with uncertainty estimates could recover additional recall without admitting more noise.
  • The co-evolution principle should transfer to other imaging domains with paired free-text reports, such as mammography or CT, where the same ratio of scarce box annotations to abundant reports holds.
  • One testable consequence of the paper's logic is that detection performance should keep improving as more report-only data is added even when box-annotated data stays fixed; this could be verified by scaling the weakly labeled pool.
  • The failure cases on overlapping and tiny lesions suggest the mutual filtering can lock in shared blind spots; a variant that periodically re-injects fully supervised training on hard cases might prevent that lock-in.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes CoE-DG, a co-evolutionary framework that jointly trains an abnormality detector and a radiology report generator on chest X-rays, using a small fully labeled set (boxes plus reports) and a large weakly labeled set (reports only). The method alternates two information-propagation mechanisms: GIP, in which the generator's predictions and features refine the detector's pseudo labels and input features, and DIP, in which the detector's boxes and categories condition the generator and provide auxiliary classification supervision. The authors report state-of-the-art results on MS-CXR and PD-CXR for detection (mAP at several IoU thresholds) and for report generation (BLEU/METEOR/ROUGE/AUC) with three generation backbones, together with ablations, a semi-oracle comparison, normal-case validation, convergence analysis, and computational-cost measurements. The central claim is that the two tasks mutually improve each other, enabling both accurate abnormality localization and clinically useful report generation from mostly weakly labeled data.

Significance. If the claims hold, the paper makes a practically valuable contribution: it shows that the abundant report-only supervision in hospital archives can be exploited to improve a box-level detector, and that detector outputs can in turn improve report generation, without requiring large paired box-and-report datasets. The manuscript is strengthened by extensive experiments and ablations on two public datasets, a semi-oracle experiment that upper-bounds the GIP filtering approximation, an explicit normal-case evaluation, release of code, and a clear comparison with several strong baselines. The main risk is that the mutual-bootstrap design has an asymmetric error-propagation path that can actively reinforce missed abnormalities, and the detection-side SOTA claim currently rests on single-run mAP numbers without significance testing.

major comments (2)
  1. [III.F, Eqs. (7)-(8)] The no-detection handling introduces a systematically biased negative supervision signal into the GIP loop. When the student detector finds no box on a weakly labeled image, the whole image is treated as a background pseudo-box (y=0, B=(0,0,W,H)) and fed to the generator; through the L_R^cls term in Eq. (7), the class token is trained to predict 'no abnormality' regardless of whether the paired report actually describes findings. Because Eq. (8) retains only detector pseudo-labels whose categories appear in the generator's predicted abnormality set, this biased class token can suppress newly proposed correct boxes in the next co-evolution iteration. This is a concrete asymmetric error-propagation path for shared blind spots, not merely generic pseudo-label noise. The semi-oracle experiment (Table III, 38.05 vs 37.89 mAP@0.5) and the failure cases in Fig. 9 are consistent with this mechanism but do not isolate it. Please quantify how often the no-detection path co-occurs with report-described abnormalities, and provide an ablation that replaces y_I=0 with report-derived positive supervision (or otherwise breaks the negative feedback) to show the framework is robust to this path.
  2. [IV.D, Table III] Detection results are reported as single mAP values without variance or significance testing, although the claimed SOTA margins over Soft Teacher are 2.07, 1.30, and 1.15 percentage points at IoU thresholds 0.25, 0.5, and 0.75 on MS-CXR. Given the known sensitivity of pseudo-label semi-supervised detection to initialization and seed, please report mean plus/minus standard deviation over at least three runs and apply a paired statistical test (for example, the Wilcoxon signed-rank test over per-test-image scores, as already used for report generation in Section IV.A) to support the central claim that CoE-DG outperforms the compared detection methods.
minor comments (6)
  1. [III.E.2] The sentence introducing Eq. (8) says 'given a weakly labeled image x in Dl'; this should be Dw, since GIP is applied to weakly labeled images.
  2. [III.E.1 and III.E.2] In DIP, the projection network f_R_prj is defined on image patches cropped according to detected boxes, but in GIP the same notation is used as f_R_prj(x) on the full image; please clarify the input resolution and architecture used for the whole-image embedding to ensure reproducibility.
  3. [IV.D] In the observations paragraph after Table III, 'MX-CXR' should be 'MS-CXR'.
  4. [IV.F, Tables VII-VIII] The normal-case evaluation is based on 30 randomly selected cases, and the reported TNR differences (10% and 3.33%) correspond to only a few cases; please provide a confidence interval or justify the sample size.
  5. [IV.E] The comparison with LLaVA-Med, CheXagent, and XrayGPT uses officially released pretrained models for inference without fine-tuning on MIMIC-CXR, so their absolute numbers are not directly comparable with the task-specific models trained on the target datasets; this limitation should be stated more prominently.
  6. [Throughout] The phrase 'language-efficiency metrics' is used in several places (e.g., Section IV.E and Section V); 'language-efficacy' or 'language quality' would be more accurate and consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the co-evolution loop is explicit, each model has independent external supervision, and the central claims are validated on held-out benchmarks.

full rationale

The paper's central claim is empirical: CoE-DG improves detection and report generation on held-out test sets relative to external SOTA baselines. The GIP/DIP alternation is an acknowledged mutual-dependence loop (Section III.E.3) rather than a hidden reduction: DIP supervises the generator with detector outputs plus ground-truth reports, since Eq. (7) includes the report log-likelihood term alongside the classification loss, and GIP filters detector pseudo-labels with the generator's class-token prediction, which is trained on both detector pseudo-labels and the paired report text. The semi-oracle experiment (Table III) and ablations (Tables IX and X) provide external checks that the cross-task signal carries information beyond the detector's own pseudo-labels, and the close semi-oracle performance shows the generator's filtering is not merely reproducing the detector's labels. The only self-citation, reference [21], is presented as a preliminary baseline and comparison point, not as load-bearing justification for the framework. No equation reduces to its own input by construction, and no fitted parameter is renamed as a prediction. The reported improvements are measured on independent test splits, so the derivation is self-contained against external benchmarks.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The framework introduces no physical entities. It does introduce two learned input representations (abnormality tokens and location embeddings) and a class-token classification head, but these are architectural components with ablation support in Tables IX and X, not ungrounded postulates. The free parameters above are all chosen on validation data or by hand, and the four listed axioms are the main unproved premises on which the co-evolution loop depends.

free parameters (4)
  • Pseudo-label confidence threshold = 0.9
    Selected on MS-CXR validation (Table II); used to retain teacher/student detection pseudo-labels in Eq. (1) and DIP detector pseudo-labels for weakly labeled images.
  • Number of co-evolution iterations and selected checkpoints = 3 iterations for detection, 4 for generation; 2nd-iteration detector and 3rd-iteration generator chosen
    Selected based on validation curves in Fig. 3; test comparisons use these chosen checkpoints.
  • Labeled-to-weakly-labeled mini-batch ratio = 2:1
    Empirically set in Section IV-B without a sensitivity study.
  • Maximum abnormality token count and NULL padding = 5
    Set to the maximum number of boxes in the training set so all inputs have consistent token length; a design constant rather than a fitted parameter.
assumptions (4)
  • domain assumption The class-token prediction of the report generator is a reliable denoising signal for the detector's pseudo-labels; only pseudo-labels whose categories appear in the generator's prediction are kept.
    Invoked in Eq. (8) (Section III-E.2); if the generator has blind spots, true boxes are discarded.
  • domain assumption The detector's pseudo-labels for weakly labeled images, after thresholding at 0.9, are reliable enough to supervise the report generator's abnormality classification and input tokens.
    Invoked in DIP (Section III-E.1) and Eq. (7); detector misses or false positives propagate into the generator.
  • domain assumption For images where neither teacher nor student detects an abnormality, or the generator predicts none, the image is excluded from the semi-supervised detection loss; this assumes these negative predictions are correct.
    Section III-F; protects against false positives but can suppress true positives when both models fail.
  • domain assumption Radiology reports accurately reflect visible abnormalities and do not omit them, so training the generator on reports transfers reliable category knowledge to the detector.
    The whole GIP scheme relies on reports being consistent with image content; realistic but not guaranteed for noisy clinical reports.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unlocking the Potential of Weakly Labeled Data: A Co-Evolutionary Learning Framework for Abnormality Detection and Report Generation." pith.science (2026). https://pith.science/paper/T5IBZUV3

@misc{pith2026241213599,
  author       = {Pith},
  title        = {Pith review of: Unlocking the Potential of Weakly Labeled Data: A Co-Evolutionary Learning Framework for Abnormality Detection and Report Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T5IBZUV3}},
  note         = {Machine review of arXiv:2412.13599}
}
read the original abstract

Anatomical abnormality detection and report generation of chest X-ray (CXR) are two essential tasks in clinical practice. The former aims at localizing and characterizing cardiopulmonary radiological findings in CXRs, while the latter summarizes the findings in a detailed report for further diagnosis and treatment. Existing methods often focused on either task separately, ignoring their correlation. This work proposes a co-evolutionary abnormality detection and report generation (CoE-DG) framework. The framework utilizes both fully labeled (with bounding box annotations and clinical reports) and weakly labeled (with reports only) data to achieve mutual promotion between the abnormality detection and report generation tasks. Specifically, we introduce a bi-directional information interaction strategy with generator-guided information propagation (GIP) and detector-guided information propagation (DIP). For semi-supervised abnormality detection, GIP takes the informative feature extracted by the generator as an auxiliary input to the detector and uses the generator's prediction to refine the detector's pseudo labels. We further propose an intra-image-modal self-adaptive non-maximum suppression module (SA-NMS). This module dynamically rectifies pseudo detection labels generated by the teacher detection model with high-confidence predictions by the student.Inversely, for report generation, DIP takes the abnormalities' categories and locations predicted by the detector as input and guidance for the generator to improve the generated reports.

Figures

Figures reproduced from arXiv: 2412.13599 by the authors.

Figure 1
Figure 1. Overview of the proposed framework. NMS: non-maximum suppression; GIP: generator-guided information propagation; DIP: detector￾guided information propagation. For semi-supervised abnormality detection, a self-adaptive NMS module dynamically rectifies pseudo detection labels generated by the teacher detection model F I t with high-confidence predictions by the student F I s . The GIP takes the feature extracted by th… view at source ↗
Figure 2
Figure 2. Illustration of the co-evolution strategy. “D” and “G” represent detector-guided and generator-guided information propagation (DIP and GIP), respectively. The k th iteration student detection model F I s,k is distilled from the teacher F I t,k−1 guided by the generation model F R k via GIP. Subsequently, F I s,k is frozen and used to 1) guide the training of the (k + 1)th generation model F R k+1 via DIP, and 2) ser… view at source ↗
Figure 3
Figure 3. Performance of the detection and report generation models as a function of iterations on the validation set of MS-CXR. under the curve (AUC) using report-extracted abnormalities by CheXBert [62]. B. Implementation The PyTorch [63] framework (1.4.0) is used for experi￾ments. Our results are obtained on a single machine with four NVIDIA V100 GPUs. For abnormality detection, we employ RetinaNet [23] with ResNet-101 [64… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Visualization of the detection results by Soft Teacher [10], GLIPv2 [67] and ours. The green and red boxes are the ground truth (GT) and predictions, respectively. The texts below images indicate abnormality categories of the boxes from left to right, and “-” indicates…
Figure 5
Figure 5. Figure 5: Example reports generated by R2Gen [7], GLIPv2 [67], our method, and the ground truth (GT). Texts highlighted with background colors are abnormalities described in GT. Texts in red indicate false positives, and those in other colors indicate false negatives. results va…
Figure 6
Figure 6. Figure 6: Training loss (LI ) and validation mAP@0.50 of the detection model, using 25%, 50%, and 100% of the fully labeled data of MS￾CXR. The plot shows the mean curves (the central dark lines) of three co-evolving iterations (iterations 0, 1, and 2 in [PITH_FULL_IMAGE:figure…
Figure 8
Figure 8. Figure 8: Examples demonstrating the results before and after applying the proposed self-adaptive NMS (SA-NMS). The prediction probabilities are overlaid on the bounding boxes (best viewed zoomed in). The SA￾NMS keeps the more confident predictions by the student model, which ar…
Figure 9
Figure 9. Figure 9: Visualization of failure detection cases by our framework. The green and red boxes are the ground truth (GT) and predictions, respectively. the pseudo detection labels as the training continues. In addition, the semi-supervised methods did not make use of the accompany…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

74 extracted references · 53 canonical work pages

  1. [1]

    Computer-aided detection in chest radiography based on artificial intelligence: A survey,

    C. Qin, D. Yao, Y . Shi, and Z. Song, “Computer-aided detection in chest radiography based on artificial intelligence: A survey,” Biomed. Eng. Online , vol. 17, no. 1, pp. 1–23, 2018

    work page 2018

  2. [2]

    Deep learning in generating radiology reports: A survey,

    M. M. A. Monshi, J. Poon, and V . Chung, “Deep learning in generating radiology reports: A survey,” Artif. Intell. in Medicine , vol. 106, p. 101878, 2020

    work page 2020

  3. [3]

    Deep learning at chest radiography: Au- tomated classification of pulmonary tuberculosis by using convolutional neural networks,

    P. Lakhani and B. Sundaram, “Deep learning at chest radiography: Au- tomated classification of pulmonary tuberculosis by using convolutional neural networks,” Radiology, vol. 284, no. 2, pp. 574–582, 2017

    work page 2017

  4. [4]

    Lung nodule detection in X-ray images: a new feature set,

    B. B. O ˘gul, P. Kos ¸ucu, A. ¨Ozc ¸am, and S. D. Kanik, “Lung nodule detection in X-ray images: a new feature set,” in Eur . Conf. IFMBE . Springer, 2015, pp. 150–155

    work page 2015

  5. [5]

    CheXNet: Radiologist-level pneumonia detection on chest X-rays with deep learning,

    P. Rajpurkar et al. , “CheXNet: Radiologist-level pneumonia detection on chest X-rays with deep learning,” arxiv Preprint arxiv:1711.05225 , 2017

    arXiv 2017

  6. [6]

    Exploring and distilling posterior and prior knowledge for radiology report generation,

    F. Liu, X. Wu, S. Ge, W. Fan, and Y . Zou, “Exploring and distilling posterior and prior knowledge for radiology report generation,” inCVPR, 2021, pp. 13 753–13 762

    work page 2021

  7. [7]

    Generating radiology reports via memory-driven transformer,

    Z. Chen, Y . Song, T.-H. Chang, and X. Wan, “Generating radiology reports via memory-driven transformer,” in EMNLP, 2020, pp. 1439– 1449

    work page 2020

  8. [8]

    Label matching semi-supervised object detection,

    B. Chen et al. , “Label matching semi-supervised object detection,” in CVPR, 2022, pp. 14 381–14 390

    work page 2022

Show all 74 references
  1. [9]

    A simple semi-supervised learning framework for object detection,

    K. Sohn, Z. Zhang, C.-L. Li, H. Zhang, C.-Y . Lee, and T. Pfister, “A simple semi-supervised learning framework for object detection,” arxiv Preprint arxiv:2005.04757, 2020

    2005 arXiv

  2. [10]

    End-to-end semi-supervised object detection with soft teacher,

    M. Xu et al. , “End-to-end semi-supervised object detection with soft teacher,” in CVPR, 2021, pp. 3060–3069

    2021

  3. [11]

    Distilling the knowledge in a neural network,

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arxiv Preprint arxiv:1503.02531 , 2015

    2015 arXiv

  4. [12]

    AlignTransformer: Hierarchical alignment of visual regions and disease tags for medical report generation,

    D. You, F. Liu, S. Ge, X. Xie, J. Zhang, and X. Wu, “AlignTransformer: Hierarchical alignment of visual regions and disease tags for medical report generation,” in MICCAI. Springer, 2021, pp. 72–82

    2021

  5. [13]

    Crossing the format boundary of text and boxes: Towards unified vision-language modeling,

    Z. Yang et al., “Crossing the format boundary of text and boxes: Towards unified vision-language modeling,” arxiv Preprint arXiv:2111.12085 , 2021

    2021 arXiv

  6. [14]

    Unifying vision-and-language tasks via text generation,

    J. Cho, J. Lei, H. Tan, and M. Bansal, “Unifying vision-and-language tasks via text generation,” in ICLR. PMLR, 2021, pp. 1931–1942

    2021

  7. [15]

    Towards general purpose vision systems: An end-to-end task-agnostic vision-language architecture,

    T. Gupta, A. Kamath, A. Kembhavi, and D. Hoiem, “Towards general purpose vision systems: An end-to-end task-agnostic vision-language architecture,” in CVPR, 2022, pp. 16 399–16 409

    2022

  8. [16]

    UniT: Multimodal multitask learning with a unified Transformer,

    R. Hu and A. Singh, “UniT: Multimodal multitask learning with a unified Transformer,” in CVPR, 2021, pp. 1439–1449

    2021

  9. [17]

    Rich feature hierarchies for accurate object detection and semantic segmentation,

    R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in CVPR, 2014, pp. 580–587

    2014

  10. [18]

    Making the most of text semantics to improve biomedical vision–language processing,

    B. Boecking et al. , “Making the most of text semantics to improve biomedical vision–language processing,” in ECCV. Springer, 2022, pp. 1–21

    2022

  11. [19]

    Weakly supervised one-stage vision and language disease detection using large scale pneumonia and pneumothorax studies,

    L. K. Tam, X. Wang, E. Turkbey, K. Lu, Y . Wen, and D. Xu, “Weakly supervised one-stage vision and language disease detection using large scale pneumonia and pneumothorax studies,” in MICCAI. Springer, 2020, pp. 45–55

    2020

  12. [20]

    MIMIC-CXR-JPG-chest radiographs with structured labels,

    A. Johnson et al., “MIMIC-CXR-JPG-chest radiographs with structured labels,” PhysioNet, 2019

    2019

  13. [21]

    You’ve got two teachers: Co-evolutionary image and report distillation for semi-supervised anatomical abnormality detection in chest X-ray,

    J. Sun et al. , “You’ve got two teachers: Co-evolutionary image and report distillation for semi-supervised anatomical abnormality detection in chest X-ray,” in MICCAI. Springer, 2023, pp. 363–373

    2023

  14. [22]

    You only look once: Unified, real-time object detection,

    J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in CVPR, 2016, pp. 779–788

    2016

  15. [23]

    Focal loss for dense object detection,

    T.-Y . Lin, P. Goyal, R. Girshick, K. He, and P. Doll ´ar, “Focal loss for dense object detection,” in ICCV, 2017, pp. 2980–2988

    2017

  16. [24]

    Fast R-CNN,

    R. Girshick, “Fast R-CNN,” in ICCV, 2015, pp. 1440–1448

    2015

  17. [25]

    Mask R-CNN,

    K. He, G. Gkioxari, P. Doll ´ar, and R. Girshick, “Mask R-CNN,” in ICCV, 2017, pp. 2961–2969

    2017

  18. [26]

    3rd place solution for the 2018 RSNA Pneumonia Detection Challenge,

    P. Cheng, “3rd place solution for the 2018 RSNA Pneumonia Detection Challenge,” https://github.com/pmcheng/rsna-pneumonia, 2019

    2018

  19. [27]

    Self-EMD: Self-supervised object detection without ImageNet,

    S. Liu, Z. Li, and J. Sun, “Self-EMD: Self-supervised object detection without ImageNet,” arxiv Preprint arXiv:2011.13677 , 2020

    2011 arXiv

  20. [28]

    Weakly supervised object localization and detection: A survey,

    D. Zhang, J. Han, G. Cheng, and M.-H. Yang, “Weakly supervised object localization and detection: A survey,” IEEE TPAMI, vol. 44, no. 9, pp. 5866–5885, 2021

    2021

  21. [29]

    Comprehensive atten- tion self-distillation for weakly-supervised object detection,

    Z. Huang, Y . Zou, B. Kumar, and D. Huang, “Comprehensive atten- tion self-distillation for weakly-supervised object detection,” NeurIPS, vol. 33, pp. 16 797–16 807, 2020

    2020

  22. [30]

    High-quality proposals for weakly supervised object detection,

    G. Cheng, J. Yang, D. Gao, L. Guo, and J. Han, “High-quality proposals for weakly supervised object detection,” IEEE TIP , vol. 29, pp. 5794– 5804, 2020. SUN et al.: CO-EVOLUTIONARY LEARNING FRAMEWORK FOR ABNORMALITY DETECTION AND REPORT GENERATION 15

    2020

  23. [31]

    Learning deep features for discriminative localization,

    B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, “Learning deep features for discriminative localization,” in CVPR, 2016, pp. 2921– 2929

    2016

  24. [32]

    Improving pneumonia localization via cross- attention on medical images and reports,

    R. Bhalodia et al. , “Improving pneumonia localization via cross- attention on medical images and reports,” in MICCAI. Springer, 2021, pp. 571–581

    2021

  25. [33]

    Anatomy- guided weakly-supervised abnormality localization in chest X-rays,

    K. Yu, S. Ghosh, Z. Liu, C. Deible, and K. Batmanghelich, “Anatomy- guided weakly-supervised abnormality localization in chest X-rays,” in MICCAI. Springer, 2022, pp. 658–668

    2022

  26. [34]

    What’s the point: Semantic segmentation with point supervision,

    A. Bearman, O. Russakovsky, V . Ferrari, and L. Fei-Fei, “What’s the point: Semantic segmentation with point supervision,” in ECCV. Springer, 2016, pp. 549–565

    2016

  27. [35]

    Point beyond class: A benchmark for weakly semi- supervised abnormality localization in chest X-rays,

    H. Ji et al. , “Point beyond class: A benchmark for weakly semi- supervised abnormality localization in chest X-rays,” in MICCAI. Springer, 2022, pp. 249–260

    2022

  28. [36]

    Consistency-based semi- supervised learning for object detection,

    J. Jeong, S. Lee, J. Kim, and N. Kwak, “Consistency-based semi- supervised learning for object detection,” NeurIPS, vol. 32, pp. 10 759– 10 768, 2019

    2019

  29. [37]

    Proposal learning for semi-supervised object detection,

    P. Tang, C. Ramaiah, Y . Wang, R. Xu, and C. Xiong, “Proposal learning for semi-supervised object detection,” in WACV, 2021, pp. 2291–2301

    2021

  30. [38]

    On the automatic generation of medical imaging reports,

    B. Jing, P. Xie, and E. Xing, “On the automatic generation of medical imaging reports,” arxiv Preprint arXiv:1711.08195 , 2017

    2017 arXiv

  31. [39]

    Multimodal recurrent model with attention for automated radiology report generation,

    Y . Xue et al., “Multimodal recurrent model with attention for automated radiology report generation,” in MICCAI. Springer, 2018, pp. 457–466

    2018

  32. [40]

    Automatic radiology report generation based on multi-view image fusion and medical concept enrichment,

    J. Yuan, H. Liao, R. Luo, and J. Luo, “Automatic radiology report generation based on multi-view image fusion and medical concept enrichment,” in MICCAI. Springer, 2019, pp. 721–729

    2019

  33. [41]

    PromptMRG: Diagnosis-driven prompts for medical report generation,

    H. Jin, H. Che, Y . Lin, and H. Chen, “PromptMRG: Diagnosis-driven prompts for medical report generation,” in AAAI, vol. 38, no. 3, 2024, pp. 2607–2615

    2024

  34. [42]

    Learning transferable visual models from natural language supervision,

    A. Radford et al. , “Learning transferable visual models from natural language supervision,” in ICML. PMLR, 2021, pp. 8748–8763

    2021

  35. [43]

    BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs,

    S. Zhang et al. , “BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs,” arXiv Preprint arXiv:2303.00915, 2023

    2023 arXiv

  36. [44]

    GPT-4 technical report,

    J. Achiam et al. , “GPT-4 technical report,” arXiv Preprint arXiv:2303.08774, 2023

    2023 arXiv

  37. [45]

    LLaMA: Open and efficient foundation language models,

    H. Touvron et al. , “LLaMA: Open and efficient foundation language models,” arXiv Preprint arXiv:2302.13971 , 2023

    2023 arXiv

  38. [46]

    Domain-specific language model pretraining for biomed- ical natural language processing,

    Y . Gu et al. , “Domain-specific language model pretraining for biomed- ical natural language processing,” ACM Trans. Comput. Healthcare , vol. 3, no. 1, pp. 1–23, 2021

    2021

  39. [47]

    BioBERT: a pre-trained biomedical language represen- tation model for biomedical text mining,

    J. Lee et al. , “BioBERT: a pre-trained biomedical language represen- tation model for biomedical text mining,” Bioinf., vol. 36, no. 4, pp. 1234–1240, 2020

    2020

  40. [48]

    Flamingo: a visual language model for few-shot learning,

    J.-B. Alayrac et al. , “Flamingo: a visual language model for few-shot learning,” NeurIPS, vol. 35, pp. 23 716–23 736, 2022

    2022

  41. [49]

    CogVLM: Visual expert for pretrained language models,

    W. Wang et al. , “CogVLM: Visual expert for pretrained language models,” arXiv Preprint arXiv:2311.03079 , 2023

    2023 arXiv

  42. [50]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,”NeurIPS, vol. 36, 2024

    2024

  43. [51]

    LLaV A-Med: Training a large language-and-vision assistant for biomedicine in one day,

    C. Li et al., “LLaV A-Med: Training a large language-and-vision assistant for biomedicine in one day,” NeurIPS, vol. 36, pp. 28 541–28 564, 2024

    2024

  44. [52]

    XRayGPT: Chest radiographs summarization using medical vision-language models,

    O. Thawkar et al. , “XRayGPT: Chest radiographs summarization using medical vision-language models,” arXiv Preprint arXiv:2306.07971 , 2023

    2023 arXiv

  45. [53]

    CheXagent: Towards a foundation model for chest x-ray interpretation,

    Z. Chen et al., “CheXagent: Towards a foundation model for chest x-ray interpretation,” in AAAI, 2024

    2024

  46. [54]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy et al. , “An image is worth 16x16 words: Transformers for image recognition at scale,” in ICLR, 2020

    2020

  47. [55]

    Cross-modal memory networks for radiology report generation,

    Z. Chen, Y . Shen, Y . Song, and X. Wan, “Cross-modal memory networks for radiology report generation,” arxiv Preprint arXiv:2204.13258, 2022

    2022 arXiv

  48. [56]

    Born again neural networks,

    T. Furlanello, Z. Lipton, M. Tschannen, L. Itti, and A. Anandkumar, “Born again neural networks,” in ICLR, 2018, pp. 1607–1616

    2018

  49. [57]

    MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs,

    A. E. Johnson et al. , “MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs,” arxiv Preprint arxiv:1901.07042, 2019

    1901 arXiv

  50. [58]

    The PASCAL visual object classes (VOC) challenge,

    M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zis- serman, “The PASCAL visual object classes (VOC) challenge,” Int. J. Comput. Vis., vol. 88, pp. 303–308, 2009

    2009

  51. [59]

    BLEU: A method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: A method for automatic evaluation of machine translation,” in Proc. 40th Annu. Meeting Assoc. for Comput. Linguistics , 2002, pp. 311–318

    2002

  52. [60]

    METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,

    S. Banerjee and A. Lavie, “METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,” in Proc. ACL Workshop on Intrinsic and Extrinsic Eval. Measures for Mach. Transl. and/or Summarization , 2005, pp. 65–72

    2005

  53. [61]

    ROUGE: A package for automatic evaluation of summaries,

    C.-Y . Lin, “ROUGE: A package for automatic evaluation of summaries,” in Text Summarization Branches Out , 2004, pp. 74–81

    2004

  54. [62]

    CheXbert: Combining automatic labelers and expert annotations for accurate radiology report labeling using BERT,

    A. Smit, S. Jain, P. Rajpurkar, A. Pareek, A. Y . Ng, and M. P. Lun- gren, “CheXbert: Combining automatic labelers and expert annotations for accurate radiology report labeling using BERT,” arxiv Preprint arXiv:2004.09167, 2020

    2004 arXiv

  55. [63]

    PyTorch: An imperative style, high-performance deep learning library,

    A. Paszke et al., “PyTorch: An imperative style, high-performance deep learning library,” NeurIPS, vol. 32, pp. 8026–8037, 2019

    2019

  56. [64]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in CVPR, 2016, pp. 770–778

    2016

  57. [65]

    ImageNet classification with deep convolutional neural networks,

    A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” Communications of the ACM , vol. 60, no. 6, pp. 84–90, 2017

    2017

  58. [66]

    Feature pyramid networks for object detection,

    T.-Y . Lin, P. Doll´ar, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in CVPR, 2017, pp. 2117–2125

    2017

  59. [67]

    GLIPv2: Unifying localization and vision-language understanding,

    H. Zhang et al. , “GLIPv2: Unifying localization and vision-language understanding,” NeurIPS, vol. 35, pp. 36 067–36 080, 2022

    2022

  60. [68]

    ORGAN: Observation- guided radiology report generation via tree reasoning,

    W. Hou, K. Xu, Y . Cheng, W. Li, and J. Liu, “ORGAN: Observation- guided radiology report generation via tree reasoning,” in AMACL (V olume 1: Long Papers), 2023, pp. 8108–8122

    2023

  61. [69]

    EV A-CLIP: Improved training techniques for CLIP at scale,

    Q. Sun, Y . Fang, L. Wu, X. Wang, and Y . Cao, “EV A-CLIP: Improved training techniques for CLIP at scale,” arXiv Preprint arXiv:2303.15389, 2023

    2023 arXiv

  62. [70]

    MedCLIP: Contrastive learning from unpaired medical images and text,

    Z. Wang, Z. Wu, D. Agarwal, and J. Sun, “MedCLIP: Contrastive learning from unpaired medical images and text,” in EMNLP, 2022, pp. 3876–3887

    2022

  63. [71]

    Automatic bounding box annotation of chest X-ray data for localization of abnormalities,

    J. Wu et al. , “Automatic bounding box annotation of chest X-ray data for localization of abnormalities,” in ISBI, 2020, pp. 799–803

    2020

  64. [72]

    METransformer: Radiology report generation by Transformer with multiple learnable expert tokens,

    Z. Wang, L. Liu, L. Wang, and L. Zhou, “METransformer: Radiology report generation by Transformer with multiple learnable expert tokens,” in CVPR, 2023, pp. 11 558–11 567

    2023

  65. [73]

    Towards open world object detection,

    K. Joseph, S. Khan, F. S. Khan, and V . N. Balasubramanian, “Towards open world object detection,” in CVPR, 2021, pp. 5830–5840

    2021

  66. [74]

    Towards open- set object detection and discovery,

    J. Zheng, W. Li, J. Hong, L. Petersson, and N. Barnes, “Towards open- set object detection and discovery,” in CVPR, 2022, pp. 3961–3970

    2022

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.