Pith. sign in

REVIEW 3 major objections 6 minor 45 references

A report-grounded vision-language foundation model for colonoscopy from 280000 routine reports

T0 review · 3 major / 6 minor · reviewed 2026-07-31 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Recovering which colonoscopy frames match which report sentences turns routine procedure notes into scalable training signal for a vision–language model that beats general biomedical encoders on retrieval, classification, and structured rep

desk verdict Solid applied VL paper: progressive report-to-frame recovery is the real contribution, and the multi-centre results mostly carry it despite same-archive retrieval caveats. read the letter →

arxiv 2607.28466 v1 pith:UAZMR77K submitted 2026-07-30 cs.AI

Jia Yu , Yan Zhu , Yili He , Zilong Wang , Xinyang Jiang , Peiyao Fu , Ruijie Yang , Tianyi Chen
show 7 more authors
Siyuan Li Zhihua Wang Fei Wu Quanlin Li Xian Yang Pinghong Zhou Shuo Wang
This is my paper · ORCID
classification cs.AI
keywords colonoscopymedicalfoundationmodelvision-languagecontrastivelearningfinding-to-framecorrespondencezero-shotclassificationstructuredreportgenerationEndoCLIPlesion-levelretrieval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Colonoscopy reports already describe lesion look, size, and location, but they summarise whole procedures, not individual frames, so the clinical text is only weakly tied to the pictures. This paper argues that you can recover those missing finding-to-frame links in stages—first finding polyp-bearing frames, then anchoring single-lesion descriptions, then disambiguating multi-lesion cases—and train a dual encoder (EndoCLIP) on the resulting 125,756 pairs drawn from about 280,000 routine records. The claim is that this recovered supervision is enough for the model to outperform general-purpose and biomedical vision–language encoders on lesion-level image–text retrieval, six multi-centre clinical classification tasks in zero-shot and linear-probe regimes, and structured report generation from frozen visual features. On benign-versus-malignant reading, a linear probe on the frozen encoder approaches expert endoscopists in a blinded comparison with twelve readers. If that holds, clinical targets can be stated in language instead of hand-labelling a new dataset for every task.

What carries the argument

Three-stage progressive correspondence recovery: case-level evidence localisation with coarse “polyp present / no finding” prompts, single-lesion anchor selection that pairs each finding sentence with its best-matching frame, and multi-lesion disambiguation that keeps sentence–frame matches only above a similarity threshold—then standard CLIP-style contrastive training on the recovered pairs.

What would settle it

A prospective, patient-grouped multi-institution test where EndoCLIP is scored on retrieval and zero-shot morphology or pathology labels drawn from centres, report styles, and scopes never seen in pretraining or EndoReport100; failure to beat the same biomedical baselines under that separation would refute the central claim.

Watch

Extended reading notes

Core claim

Progressively recovering lesion-level image–text pairs from routine colonoscopy reports yields scalable cross-modal supervision: EndoCLIP trained on 125,756 recovered pairs outperforms general-purpose and biomedical vision–language encoders on lesion-level retrieval (global mean Recall@1 14.3% versus 2.8% for the strongest comparator), six multi-centre clinical classification tasks in zero-shot and linear-probe settings, and structured report generation, with a frozen linear probe approaching expert readers on benign-versus-malignant classification.

Load-bearing premise

The automatically chosen frame–sentence matches, especially multi-lesion pairs kept only when similarity clears a hand-set threshold, are accurate and unbiased enough that later gains reflect real clinical meaning rather than archive style or self-reinforced matching errors.

Editorial extensions

If this is right

  • Clinical concepts such as Paris type, surface pattern, size threshold, or malignancy can be specified as text prompts instead of building a separate labelled classifier for each target.
  • Routine procedure documentation becomes a large, reusable training resource once finding-to-frame links are recovered, reducing dependence on dense frame-level annotation.
  • Frozen endoscopy-specific visual features can drive structured JSON drafts (diameter, Paris class, descriptor sets) for endoscopist review more accurately than general biomedical encoders under the same decoder.
  • Language-based lesion retrieval can support case review and teaching by locating frames from textual descriptions in multi-lesion procedures.
  • The same correspondence-recovery idea is proposed as transferable to other many-frame, many-finding procedure records such as upper GI endoscopy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If source identity remains readable in the embeddings, apparent multi-centre gains may partly track acquisition or reporting style; hard site-holdout and style-matched negatives would separate true semantics from site cues.
  • The largest label-efficient gains on rare descriptors (villous, lobulated) suggest the method is most valuable precisely where new hand-labelled cohorts are hardest to build.
  • A natural next measurement is calibration and uncertainty of prompt scores under real time pressure, not only rank and AUC on static benchmarks.
  • Thresholded multi-lesion matching could systematically drop hard or atypical lesions; auditing discarded pairs against expert links would show whether the training set is skewed toward easy cases.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The manuscript introduces EndoCLIP, a CLIP-style dual-encoder foundation model for colonoscopy trained on 125,756 lesion-level image–text pairs recovered from 280,476 routine reports. Because reports summarise whole procedures rather than caption frames, the authors propose a three-stage progressive correspondence-recovery pipeline (case-level evidence localisation → single-lesion anchor selection → multi-lesion disambiguation with a similarity threshold) and train with symmetric InfoNCE. They release EndoReport100 for lesion-level retrieval and evaluate on global/intra-case retrieval, six multi-centre classification tasks (zero-shot, linear probe, end-to-end), embedding kNN structure, structured report generation with a frozen Qwen3-14B decoder, stage ablations, a naive case-level pairing control, and a blinded 12-endoscopist pathology reader study. EndoCLIP outperforms general-purpose and biomedical VL encoders on these benchmarks, with a frozen linear probe approaching expert-tier accuracy on benign-versus-malignant classification.

Significance. If the recovered pairs largely reflect true finding-to-frame clinical semantics, the work is a substantial contribution: it shows that routine endoscopy documentation can supply scalable vision–language supervision without per-task dense annotation, and that clinical targets can be specified in language. Strengths that should be credited include the multi-protocol evaluation design (bootstrap CIs and paired tests on EndoReport100; multi-seed label-efficiency curves; frozen-decoder structured reporting), the stage-wise ablation separating coarse polyp detection from descriptive retrieval, the naive case-level control, public release of EndoReport100 and code, and the blinded reader comparison. These make the empirical package stronger than typical medical VL pretraining reports and give the field a concrete benchmark for multi-lesion report-grounded retrieval.

major comments (3)
  1. [Methods §4.4.2; Supp. Fig. S6 / Table S22] Methods §4.4.2 (Stage III): multi-lesion pairs (~56.2k of 125.8k) are retained only when EndoCLIP-III similarity exceeds a fixed threshold of 0.28 chosen on a held-out multi-lesion validation set (Supp. Fig. S6, Table S22). The central claim—that recovering finding-to-frame correspondence yields scalable clinical supervision—depends on these matches being sufficiently correct and unbiased. The manuscript does not report a human audit of match precision/error modes (wrong lesion, normal mucosa, style-correlated near-misses) at the operating threshold, nor sensitivity of EndoReport100 and EndoVL gains to that threshold. A modest clinician-rated sample of accepted/rejected multi-lesion pairs, plus a brief threshold sweep on downstream metrics, is needed to ground the axiom that model-selected pairs encode clinical semantics rather than self-reinforced similarity.
  2. [Results §2.2; Discussion; Supp. Fig. S5 / Table S18] Results §2.2 and Discussion: EndoReport100 is independent of the pretraining split but drawn from the same institutional archive, and the Discussion notes that source identity remains locally predictable in the embeddings (Supp. Fig. S5, Table S18). Global mean Recall@1 of 14.3% vs 2.8% for PMC-CLIP is a strong headline, but same-archive style or acquisition cues could inflate retrieval and some transfer without proving lesion-level semantic correspondence. The naive case-level control (global R@1 6.5%) shows progressive recovery helps, yet does not separate true correspondence quality from archive memorization. Please either (i) quantify how much retrieval/classification signal survives controls that disrupt clinical text while preserving style (e.g., shuffled descriptors within site, or site-only prompts), or (ii) more tightly bound claims that rest primarily on EndoReport100 versus th
  3. [Results §2.3; Methods §4.1–4.3; Supp. Table S15] Results §2.3 / reader study: the frozen EndoCLIP probe is reported to approach expert-tier accuracy on benign-versus-malignant classification (0.852±0.010 vs experts 0.846±0.014), with exploratory McNemar tests. Clarify whether the Zhongshan pathology images and patients are disjoint from the 280k-report pretraining archive (same centre; Methods §4.1–4.3). If overlap or near-overlap is possible, the reader comparison and in-house pathology AUC cannot be read as fully external. State the separation rule explicitly and, if full disjointness cannot be guaranteed, mark the reader study as same-centre and lean on EndoVL for external claims.
minor comments (6)
  1. [Fig. 1] Fig. 1 caption begins with a spaced typo: “W eak report–image alignment”. Fix.
  2. [Abstract; Fig. 1b] Abstract and main text say 125,756 pairs / 280,476 records; Fig. 1b funnel text uses 125.8k / 280.5k and 104.5k cases—align rounding and the curated-case count with Supplementary Table S1 everywhere.
  3. [Methods §4.4.3–4.4.4] Methods §4.4.3 Eq. (1): state whether τ is the learnable CLIP logit scale (as in §4.4.4) or a fixed temperature; the two subsections should use identical notation.
  4. [Table 1; Fig. 4; Supp. Table S2] Table 1 and Fig. 4: define the exact clinical prompt templates in the main text or point more visibly to Supplementary Table S2 when discussing zero-shot failures of PMC-CLIP (e.g., size sensitivity 0.002).
  5. [Fig. 6g] Fig. 6g: the claim that 91% of Paris misclassifications are adjacent on the protrusion continuum is important; add the numeric off-diagonal counts in the supplement if not already fully tabulated.
  6. [§5–6 Data/Code availability] Data/code availability statements are clear and appreciated; ensure the GitHub URL and Figshare DOIs remain stable through review and that EndoVL split manifests document any unavoidable patient-ID gaps noted in the Discussion.

Circularity Check

1 steps flagged · score 1.0 of 10

No load-bearing circularity: progressive pair recovery is self-training for data construction, while headline metrics rest on clinician-labeled and multi-centre held-out benchmarks.

  1. other [Methods §4.4.2 Stage III; also Fig. 2 progressive pipeline]
    "For every finding sentence t_j, we retrieved the most similar retained frame, x⋆_j = arg max_{x∈X̃_c} s(x, t_j). We retained (x⋆_j, t_j) only when s(x⋆_j, t_j) exceeded a fixed threshold of 0.28, selected on a separate held-out validation set of multi-lesion cases."

    Later-stage training pairs are chosen by an earlier EndoCLIP checkpoint and a threshold tuned on held-out multi-lesion cases, so a fraction of the 56.2k multi-lesion pairs is model-reinforced. This is mild self-training circularity in data construction only; it does not make EndoReport100 Recall@1, EndoVL AUCs, or reader-study accuracy equal the selection criterion by construction, because those labels are clinician- or public-benchmark-provided.

full rationale

EndoCLIP’s central claims are empirical outperformance on lesion-level retrieval (EndoReport100), six classification tasks (EndoVL public sources plus an independent pathology cohort), structured report generation, and a blinded 12-reader study. Those metrics are not algebraic restatements of the contrastive objective or of the pair-selection rule. Progressive recovery (Stages I–III) uses earlier checkpoints to propose frame–sentence pairs and a similarity threshold (0.28) chosen on a held-out multi-lesion validation set—standard self-training / hyperparameter selection, not a fitted quantity renamed as a prediction. A naive case-level pairing control (global mean Recall@1 6.5% vs 14.3%) further shows the pipeline’s contribution is measured rather than assumed. Self-citations (EndoFinder, GI-Bench, prior endoscopy work) supply background and related tools; none supply a uniqueness theorem or uniqueness-forced ansatz that the present results reduce to. Mild residual concern that multi-lesion pairs and same-archive EndoReport100 could share style cues is a generalization/validity issue, not circular derivation. Score 1 only for the self-reinforcing character of model-selected training pairs; no step reduces evaluation claims to inputs by construction.

Assumptions & free parameters 4 free parameters · 5 assumptions · 2 invented entities

The central claim rests on standard contrastive VL learning plus domain assumptions that routine report sentences can be aligned to still frames well enough to supervise clinically meaningful embeddings, and on pipeline choices (stage design, similarity threshold, generic prompts) that define which pairs count as supervision.

free parameters (4)
  • multi-lesion match similarity threshold = 0.28
    Pairs retained only if max frame–sentence cosine similarity exceeds a fixed threshold chosen on held-out multi-lesion validation; directly controls which multi-polyp pairs enter final training.
  • InfoNCE temperature τ (learnable logit scale) = initialized at 0.07 (learnable)
    Standard CLIP temperature initialized then learned; affects sharpness of contrastive alignment at every stage.
  • contrastive pretraining hyperparameters = lr 1e-5, wd 0.1, 20 epochs/stage, global batch 32
    Learning rate, weight decay, epochs, batch size, and warmup set training dynamics and final representation quality.
  • top-p / frame retention rule after Stage I = top-scoring retained frames per case (exact p as implemented in pipeline)
    Low-scoring background frames discarded using EndoCLIP-I similarities before later stages; selection policy is part of recovered supervision.
assumptions (5)
  • standard math Symmetric image–text InfoNCE on ℓ2-normalized embeddings yields a shared space where matched lesion frames and finding sentences are nearest neighbors.
    Standard CLIP/InfoNCE assumption (Methods §4.4.3, Eq. 1); not re-proved.
  • domain assumption Routine free-text finding sentences, after segmentation and controlled-vocabulary parsing, are faithful enough descriptions of visible lesions to serve as supervision.
    Load-bearing clinical NLP assumption in §4.2–4.3; report quality and completeness are not independently audited beyond curation filters.
  • domain assumption In single-polyp cases, the single finding sentence is a low-ambiguity anchor for the best-matching retained frame.
    Stage II premise (§4.4.2); fails if the still-frame set misses the described lesion or the sentence is non-visual.
  • ad hoc to paper Model-chosen argmax matches above threshold in multi-lesion cases recover true finding-to-frame correspondence often enough to improve semantics.
    Core of Stage III; validated indirectly via ablations and naive-pairing control, not by large-scale human verification of all recovered pairs.
  • domain assumption External EndoVL public images and the Zhongshan pathology cohort are sufficiently out-of-distribution relative to pretraining to support transfer claims.
    Stated multi-centre evaluation design; tempered by Discussion notes on source predictability and unverifiable patient-level splits for report generation.
invented entities (2)
  • EndoCLIP progressive correspondence-recovery pipeline (Stages I–III checkpoints) independent evidence
    purpose: Convert case-level report–frame bags into lesion-level image–text pairs for contrastive pretraining.
    Methodological construct defined in this paper; intermediate checkpoints EndoCLIP-I/II/III exist only to implement recovery and ablations.
  • EndoReport100 benchmark independent evidence
    purpose: Clinician-curated multi-lesion retrieval and frame-level evaluation set (100 cases, 217 lesions, 7002 frame labels).
    New evaluation resource released on Figshare; independent of pretraining pairs but same institutional archive.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A report-grounded vision-language foundation model for colonoscopy from 280000 routine reports." pith.science (2026). https://pith.science/paper/UAZMR77K

@misc{pith2026260728466,
  author       = {Pith},
  title        = {Pith review of: A report-grounded vision-language foundation model for colonoscopy from 280000 routine reports},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UAZMR77K}},
  note         = {Machine review of arXiv:2607.28466}
}
read the original abstract

Vision-language models remain underused in colonoscopy despite the rich expert descriptions recorded in routine reports. These reports document lesion appearance, size and location but summarise entire procedures rather than caption individual frames, leaving clinical findings only weakly linked to the corresponding images. Here we develop EndoCLIP, a colonoscopy vision-language foundation model trained on 125,756 lesion-level image-text pairs progressively recovered from 280,476 routine colonoscopy records. Across lesion-level image-text retrieval, structured report generation and six multi-centre clinical classification tasks, EndoCLIP outperforms general-purpose and biomedical vision-language encoders in both zero-shot and linear-probe settings. On benign-versus-malignant classification, its linear probe approaches the performance of expert readers in a blinded study involving 12 endoscopists. These results suggest that recovering finding-to-frame correspondence can transform routine documentation into scalable supervision, enabling clinical targets to be specified in language rather than separately annotated for each task.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 9 canonical work pages

  1. [1]

    CA: A Cancer Journal for Clinicians74(3), 229–263 (2024) https://doi.org/10.3322/caac.21834

    Bray, F., Laversanne, M., Sung, H., Ferlay, J., Siegel, R.L., Soerjomataram, I., Jemal, A.: Global cancer statistics 2022: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: A Cancer Journal for Clinicians74(3), 229–263 (2024) https://doi.org/10.3322/caac.21834

    doi:10.3322/caac.21834 2022

  2. [2]

    New England Journal of Medicine 329(27), 1977–1981 (1993)

    Winawer, S.J., Zauber, A.G., Ho, M.N., O’Brien, M.J., Gottlieb, L.S., Sternberg, S.S., Waye, J.D., Schapiro, M., Bond, J.H., Panish, J.F.,et al.: Prevention of colorectal cancer by colonoscopic polypectomy. New England Journal of Medicine 329(27), 1977–1981 (1993)

    1977

  3. [3]

    New England Journal of Medicine366(8), 687–696 (2012)

    Zauber, A.G., Winawer, S.J., O’Brien, M.J., Lansdorp-Vogelaar, I., Ballegooijen, M., Hankey, B.F., Shi, W., Bond, J.H., Schapiro, M., Panish, J.F.,et al.: Colono- scopic polypectomy and long-term prevention of colorectal-cancer deaths. New England Journal of Medicine366(8), 687–696 (2012)

    2012

  4. [4]

    New England Journal of Medicine387(17), 1547–1556 (2022)

    Bretthauer, M., Løberg, M., Wieszczy, P., Kalager, M., Emilsson, L., Garborg, K., Rupinski, M., Dekker, E., Spaander, M., Bugajski, M.,et al.: Effect of colonoscopy screening on risks of colorectal cancer and related death. New England Journal of Medicine387(17), 1547–1556 (2022)

    2022

  5. [5]

    Gastroenterology155(4), 1069–1078 (2018)

    Urban, G., Tripathi, P., Alkayali, T., Mittal, M., Jalali, F., Karnes, W., Baldi, P.: Deep learning localizes and identifies polyps in real time with 96% accuracy in screening colonoscopy. Gastroenterology155(4), 1069–1078 (2018)

    2018

  6. [6]

    The lancet Gastroenterology & hepatology4(1), 71–80 (2019)

    Ahmad, O.F., Soares, A.S., Mazomenos, E., Brandao, P., Vega, R., Seward, E., Stoyanov, D., Chand, M., Lovat, L.B.: Artificial intelligence and computer- aided diagnosis in colonoscopy: current evidence and future directions. The lancet Gastroenterology & hepatology4(1), 71–80 (2019)

    2019

  7. [7]

    Gut68(10), 1813–1819 (2019)

    Wang, P., Berzin, T.M., Glissen Brown, J.R., Bharadwaj, S., Becq, A., Xiao, X., Liu, P., Li, L., Song, Y., Zhang, D.,et al.: Real-time automatic detection 19 system increases colonoscopic polyp and adenoma detection rates: a prospective randomised controlled study. Gut68(10), 1813–1819 (2019)

    2019

  8. [8]

    Gastroenterology159(2), 512–520 (2020)

    Repici, A., Badalamenti, M., Maselli, R., Correale, L., Radaelli, F., Rondonotti, E., Ferrara, E., Spadaccini, M., Alkandari, A., Fugazza, A.,et al.: Efficacy of real-time computer-aided detection of colorectal neoplasia in a randomized trial. Gastroenterology159(2), 512–520 (2020)

    2020

Show all 45 references
  1. [9]

    Gastrointestinal Endoscopy97(5), 815–8241 (2023) https://doi.org/10.1016/j.gie.2022.10.016

    Parasa, S., Repici, A., Berzin, T., Leggett, C., Gross, S.A., Sharma, P.: Frame- work and metrics for the clinical use and implementation of artificial intelligence algorithms into endoscopy practice: recommendations from the american society for gastrointestinal endoscopy art...

    2023 doi

  2. [10]

    Medical Image Analysis 98, 103298 (2024) https://doi.org/10.1016/j.media.2024.103298

    Boers, T.G.W., Fockens, K.N., Putten, J.A., Jaspers, T.J.M., Kusters, C.H.J., Jukema, J.B., Jong, M.R., Struyvenberg, M.R., Groof, J., Bergman, J.J., With, P.H.N., Sommen, F.: Foundation models in gastrointestinal endoscopic ai: Impact of architecture, pre-training approach an...

    2024

  3. [11]

    IEEE Transactions on Medical Imaging44(12), 5245–5257 (2025)

    Yu, J., Zhu, Y., Fu, P., Chen, T., Huang, J., Li, Q., Zhou, P., Wang, Z., Wu, F., Wang, S., Yang, X.: Robust polyp detection and diagnosis through compositional prompt-guided diffusion models. IEEE Transactions on Medical Imaging44(12), 5245–5257 (2025)

    2025

  4. [12]

    Nature622(7981), 156–163 (2023) https://doi.org/10.1038/s41586-023-06555-x

    Zhou, Y., Chia, M.A., Wagner, S.K., Ayhan, M.S., Williamson, D.J., Struyven, R.R., Liu, T., Xu, M., Lozano, M.G., Woodward-Court, P.,et al.: A foundation model for generalizable disease detection from retinal images. Nature622(7981), 156–163 (2023) https://doi.org/10.1038/s415...

    2023 doi

  5. [13]

    In: International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), pp

    Wang, Z., Liu, C., Zhang, S., Dou, Q.: Foundation model for endoscopy video analysis via large-scale self-supervised pre-train. In: International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), pp. 101–111 (2023). https://doi.org/10.1007/978-...

    2023 doi

  6. [14]

    In: Proceedings of the 38th International Conference on Machine Learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J.,et al.: Learning transfer- able visual models from natural language supervision. In: Proceedings of the 38th International Conference on Machine Learning. Proc...

    2021

  7. [15]

    In: Proceedings of the 38th International Conference on Machine Learning

    Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q.V., Sung, Y.-H., Li, Z., Duerig, T.: Scaling up visual and vision- language representation learning with noisy text supervision. In: Proceedings of the 38th International Conference on Machine Learning. Proce...

    2021

  8. [16]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp

    Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., Jitsev, J.: Reproducible scaling laws for con- trastive language-image learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CV...

    2023

  9. [17]

    In: Machine Learning for Healthcare Conference (MLHC) (2022)

    Zhang, Y., Jiang, H., Miura, Y., Manning, C.D., Langlotz, C.P.: Contrastive learn- ing of medical visual representations from paired images and text. In: Machine Learning for Healthcare Conference (MLHC) (2022). PMLR

    2022

  10. [18]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp

    Huang, S.-C., Shen, L., Lungren, M.P., Yeung, S.: GLoRIA: A multimodal global-local representation learning framework for label-efficient medical image recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3942–3951 (2021)

    2021

  11. [19]

    Nature Biomedical Engineering6(12), 1399–1406 (2022)

    Tiu, E., Talius, E., Patel, P., Langlotz, C.P., Ng, A.Y., Rajpurkar, P.: Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning. Nature Biomedical Engineering6(12), 1399–1406 (2022)

    2022

  12. [20]

    Nature Medicine29(9), 2307–2316 (2023)

    Huang, Z., Bianchi, F., Yuksekgonul, M., Montine, T.J., Zou, J.: A visual– language foundation model for pathology image analysis using medical Twitter. Nature Medicine29(9), 2307–2316 (2023)

    2023

  13. [21]

    Nature Medicine30(3), 863–874 (2024)

    Lu, M.Y., Chen, B., Williamson, D.F.K., Chen, R.J., Liang, I., Ding, T., Jaume, G., Odintsov, I., Le, L.P., Gerber, G.,et al.: A visual-language foundation model for computational pathology. Nature Medicine30(3), 863–874 (2024)

    2024

  14. [22]

    Nature Biomedical Engineering (2026) https://doi.org/10.1038/s41551-025-01599-y

    Hamamci, I.E., Er, S., Wang, C., Almas, F., Simsek, A.G., Esirgun, S.N., Dogan, I., Durugol, O.F., Hou, B., Shit, S.,et al.: Generalist foundation models from a multimodal dataset for 3D computed tomography. Nature Biomedical Engineering (2026) https://doi.org/10.1038/s41551-0...

    2026 doi

  15. [23]

    Nature Biomedical Engineering (2025) https://doi.org/10.1038/s41551-025-01500-x

    Wang, S., Zhu, Y., Yang, Z., Luo, X., Zhang, Y., Fu, P., Wang, H., Wang, M., Song, Z., Li, Q., Zhou, P., Guo, Y.: Leveraging large language and vision models for knowledge extraction from large-scale image–text colonoscopy records. Nature Biomedical Engineering (2025) https://...

    2025 doi

  16. [24]

    Figshare

    Li, S.: EndoReport100. Figshare. Dataset (2026). https://doi.org/10.6084/m9. figshare.33085637 . https://doi.org/10.6084/m9.figshare.33085637

    2026 doi

  17. [25]

    Figshare

    Fu, P., Li, S., Zhu, Y., Wang, S., Li, X., Liu, X., He, M., Li, Q., Zhou, P.- H.: EndoVL: An expert annotated dataset of public gastrointestinal endoscopic images with uncertainty labels. Figshare. Dataset (2026). https://doi.org/10. 6084/m9.figshare.32320998 . https://doi.org...

    2026 doi

  18. [26]

    In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2023, pp

    Lin, W., Zhao, Z., Zhang, X., Wu, C., Zhang, Y., Wang, Y., Xie, W.: Pmc-clip: Contrastive language-image pre-training using biomedical documents. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2023, pp. 525–

    2023

  19. [27]

    Gastrointestinal Endoscopy 58(6), 3–43 (2003) https://doi.org/10.1016/S0016-5107(03)02159-X

    Participants in the Paris Workshop: The Paris endoscopic classification of superfi- cial neoplastic lesions: esophagus, stomach, and colon. Gastrointestinal Endoscopy 58(6), 3–43 (2003) https://doi.org/10.1016/S0016-5107(03)02159-X

    2003 doi

  20. [28]

    Gastrointestinal Endoscopy78(4), 625–632 (2013) https://doi.org/10.1016/j.gie.2013.04.185

    Hayashi, N., Tanaka, S., Hewett, D.G., Kaltenbach, T.R., Sano, Y., Ponchon, T., Saunders, B.P., Rex, D.K., Soetikno, R.M.: Endoscopic prediction of deep sub- mucosal invasive carcinoma: validation of the narrow-band imaging international colorectal endoscopic (NICE) classifica...

    2013 doi

  21. [29]

    Gastroenterology158(4), 1131–1153 (2020) https://doi.org/10.1053/j.gastro.2019.10.026

    Gupta, S., Lieberman, D., Anderson, J.C., Burke, C.A., Dominitz, J.A., Kaltenbach, T., Robertson, D.J., Shaukat, A., Syngal, S., Rex, D.K.: Recommen- dations for follow-up after colonoscopy and polypectomy: a consensus update by the US multi-society task force on colorectal ca...

    2020 doi

  22. [30]

    NEJM AI2(1) (2025) https://doi.org/10.1056/ AIoa2400640

    Zhang, S., Xu, Y., Usuyama, N., Xu, H., Bagga, J., Tinn, R., Preston, S., Rao, R., Wei, M., et al.: A multimodal biomedical foundation model trained from fifteen million image–text pairs. NEJM AI2(1) (2025) https://doi.org/10.1056/ AIoa2400640

    2025

  23. [31]

    Clinical and Translational Gastroenterology 12(1), 00282 (2020) https://doi.org/10.14309/ctg.0000000000000282

    Qu, J.-Y., Li, Z., Su, J.-R., Ma, M.-J., Xu, C.-Q., Zhang, A.-J., Liu, C.-X., Yuan, H.-P., Chu, Y.-L., Lang, C.-C., Huang, L.-Y., Lu, L., Li, Y.-Q., Zuo, X.-L.: Devel- opment and validation of an automatic image-recognition endoscopic report gen- eration system: A multicenter ...

    2020 doi

  24. [32]

    npj Digital Medicine9(1), 394 (2026) https: //doi.org/10.1038/s41746-026-02569-7

    Jiang, R., Chen, B., Dong, Z., Zeng, X., You, H., Li, Y., Deng, Y., Mu, G., Wang, J., Huang, L., Li, J., Cheng, D., Zhou, W., Yu, H.: Domain specific multimodal large language model for automated endoscopy reporting with mul- ticenter prospective validation. npj Digital Medici...

    2026 doi

  25. [33]

    arXiv preprint arXiv:2601.08183 (2026)

    Zhu, Y., Luo, T., Fu, P.-Y., Zhang, Z., Wang, Z.-L., Qu, Y.-F., Geng, Z.-H., Xu, J.-Q., Yao, L., Ma, L.-Y., Su, W., Chen, W.-F., Li, Q.-L., Wang, S., Zhou, P.-H.: GI-Bench: A panoramic benchmark revealing the knowledge-experience dissociation of multimodal large language model...

    2026

  26. [34]

    In: Medical Image Computing and Computer Assisted Intervention 22 – MICCAI 2024

    Yang, R., Zhu, Y., Fu, P., Zhang, Y., Wang, Z., Li, Q., Zhou, P., Yang, X., Wang, S.: EndoFinder: Online Image Retrieval for Explainable Colorectal Polyp Diagnosis . In: Medical Image Computing and Computer Assisted Intervention 22 – MICCAI 2024. Lecture Notes in Computer Scie...

    2024 doi

  27. [35]

    In: International Conference on Learning Representations (ICLR) (2021)

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on L...

    2021

  28. [36]

    Advances in Neural Information Processing Systems (2017)

    Vaswani, A.: Attention is all you need. Advances in Neural Information Processing Systems (2017)

    2017

  29. [37]

    arXiv preprint arXiv:1807.03748 (2018)

    Oord, A., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)

    2018 arXiv

  30. [38]

    In: Interna- tional Conference on Learning Representations (ICLR) (2019)

    Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: Interna- tional Conference on Learning Representations (ICLR) (2019)

    2019

  31. [39]

    ACM Transactions on Computing for Healthcare3(1), 1–23 (2021)

    Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., Poon, H.: Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare3(1), 1–23 (2021)

    2021

  32. [40]

    saliency maps from physicians

    Bernal, J., S´ anchez, F.J., Fern´ andez-Esparrach, G., Gil, D., Rodr ´ ıguez, C., Vilari˜ no, F.: WM-DOV A maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Computerized Medical Imaging and Graphics43, 99–111 (2015)

    2015

  33. [41]

    Journal of Healthcare Engineering2017, 4037190 (2017)

    V´ azquez, D., Bernal, J., S´ anchez, F.J., Fern´ andez-Esparrach, G., L´ opez, A.M., Romero, A., Drozdzal, M., Courville, A.: A benchmark for endoluminal scene segmentation of colonoscopy images. Journal of Healthcare Engineering2017, 4037190 (2017)

    2017

  34. [42]

    In: International Conference on Multimedia Modeling, pp

    Jha, D., Smedsrud, P.H., Riegler, M.A., Halvorsen, P., Lange, T., Johansen, D., Johansen, H.D.: Kvasir-SEG: A segmented polyp dataset. In: International Conference on Multimedia Modeling, pp. 451–462 (2020). Springer

    2020

  35. [43]

    Scientific Data 10(1), 75 (2023)

    Ali, S., Jha, D., Ghatwary, N., Realdon, S., Cannizzaro, R., Salem, O.E., Lamar- que, D., Daul, C., Riegler, M.A., Anonsen, K.V.,et al.: A multi-centre polyp detection and segmentation dataset for generalisability assessment. Scientific Data 10(1), 75 (2023)

    2023

  36. [44]

    Journal of Machine Learning Research9, 2579–2605 (2008) 23

    Maaten, L., Hinton, G.: Visualizing data using t-SNE. Journal of Machine Learning Research9, 2579–2605 (2008) 23

    2008

  37. [536]

    https://doi.org/10.1007/978-3-031-43993-3 51

    Springer, Cham (2023). https://doi.org/10.1007/978-3-031-43993-3 51

    2023 doi

Pith tools

Reviewed July 31, 2026 · model on record in the stance chip above.