{"id":"6a3b7cf1-087e-40d5-8b26-af0de2092591","arxiv_id":"2608.04472","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"EndoVLM pre-trains an endoscopy vision-language model on unordered image-set/report pairs using anatomy-guided sparse pooling, progressive semantic alignment, and a semantic-concentrated masked autoencoder, and it outperforms prior endoscopy foundation models on several downstream tasks.","lead":"EndoVLM is a vision-language model pre-trained on 348K endoscopy exams that pair clinical reports with sets of images. It uses text-guided attention to select relevant frames, then aligns visual and text spaces in stages, and reports gains on polyp diagnosis, polyp segmentation, and zero-shot anatomy recognition. A generalist reader might care because it offers a route to endoscopy foundation models that use existing clinical reports instead of frame-level labels.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unvalidated Qwen3 parsing is the load-bearing link: AGSP queries and PSAA targets inherit every parsing error, and no clinician audit or accuracy statistic is reported.","rationale":"The reader's conditional verdict is well calibrated, and the weakest assumption identified in the reader's report is also the most load-bearing concern I find. The pre-training pipeline is a chain: Qwen3 parses reports, AGSP uses those parses as queries, PSAA uses the same parses to define anatomy and pathological-status targets, and even the dataset filtering is Qwen3-driven. A systematic error anywhere in that chain propagates to the learned visual representations, so the empirical superiority claim cannot be evaluated without a parsing audit. I do not find an internal inconsistency that would force rejection: the benchmark settings are named, the ablation supports the contribution of each component, and the admitted lower-ileum limitation is localized and plausible. The absence of error bars and the lack of released weights are rigor and reproducibility concerns, but they do not change the conditional status. The concrete test I propose is a clinician audit of a few hundred reports; if the audit passes, the paper could be upgraded to acceptance, and if it fails, the central claim collapses. Since the reader already reached CONDITIONAL and my analysis reinforces that same conclusion, the verdict should remain unchanged.","tokens_in":857,"tokens_out":794,"duration_ms":128436,"concrete_test":"Randomly sample 500 reports from the curated 348K set. Have two gastroenterologists independently annotate each report for (1) eligibility under the stated completeness criteria and (2) the semantic triplets (anatomy label, finding proposition, normal/abnormal) for every region, using the same schema given to Qwen3. Compare the clinician annotations to Qwen3 outputs, reporting per-triplet anatomy accuracy, abnormal-status precision/recall/F1, and Cohen's kappa. If abnormal-status F1 is below 0.95 or anatomy accuracy is below 0.98, the pre-training targets are materially corrupted; the authors should retrain with human-corrected labels or demonstrate through explicit label-noise robustness experiments that downstream results are unchanged. This audit alone determines whether the parser concern actually lands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that EndoVLM's pre-training yields superior downstream representations and zero-shot ability—rests on the correctness of Qwen3's parse of each clinical report into semantic triplets (finding, anatomy, pathological status). Section 2.1 states 'we use Qwen3 ... to parse T_i into semantic triplets' and Section 3 states 'By leveraging Qwen3 ... we filtered out ineligible cases.' No parsing accuracy, human-agreement statistic, or failure analysis is reported. This matters because the AGSP queries in Eq. (1) and the PSAA soft targets in Eq. (4) are built directly from these triplets: a wrong anatomical label makes z_{i,j} pool frames from the wrong region, and a wrong pathological-status label flips the supervision target between normal and abnormal. The same parser also determines which cases are 'ineligible' and excluded from the 348K corpus, so systematic parser errors could silently bias the training set, for example by dropping complex or unusual examinations. The superiority of EndoVLM is therefore conditional on an unmeasured and potentially large source of label noise, with no reported evidence that this noise is small.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EndoVLM, a vision-language pre-training model for gastrointestinal endoscopy. The model is trained on 348K examination-level pairs, where each pair is an unordered set of endoscopic images and a clinical report. Three main components are proposed: (1) Anatomy-Guided Sparse Pooling (AGSP) uses fine-grained text embeddings from the report as queries to select the top-K relevant frames per anatomical region; (2) Progressive Semantic-Aware Alignment (PSAA) combines a global image-set/report InfoNCE loss with a fine-grained contrastive loss over anatomy-pathology soft targets; (3) Semantic-Concentrated Masked Autoencoder (SC-MAE) reconstructs masked patches only on the semantically selected frames. The authors evaluate the model on polyp diagnosis, polyp segmentation generalization, ulcerative colitis severity grading, and zero-shot anatomy recognition and video polyp detection, reporting that it outperforms existing general and endoscopy-specific foundation models and is competitive with task-specific models. They also provide an ablation study showing each loss component contributes.","tokens_in":10211,"tokens_out":8876,"duration_ms":75689,"significance":"If the reported results hold, EndoVLM would be a valuable reusable backbone for endoscopic video and image-set analysis, reducing the need for frame-level annotations. The proposed use of LLM-parsed clinical reports to guide sparse frame selection is an interesting and practical direction, and the taxonomy-aware soft targets are a reasonable way to encode clinical prior knowledge. The paper provides a public code link and the ablations isolate the contribution of each component, which supports reproducibility. However, the strongest claims are conditional on the accuracy of the Qwen3-based parsing and on the fairness of the comparative protocol. The zero-shot results are presented with insufficient detail. As such, the significance is potentially high but currently not fully established.","major_comments":[{"comment":"The entire pre-training pipeline relies on Qwen3 to parse each clinical report into semantic triplets (finding, anatomy, pathological status) and to filter ineligible examinations, yet the paper reports no validation of this parsing step—no accuracy, precision/recall against clinician-annotated reports, or failure analysis. Since the AGSP queries in Eq. (1) and the PSAA soft targets in Eq. (4) are built directly from these triplets, any systematic parsing error (e.g., mislabeled anatomy or flipped pathology status) directly corrupts the visual-text alignment, and the filtering step may silently bias the 348K-exam corpus by dropping complex cases. To support the central claim, the authors should provide a human-annotated parsing audit (e.g., 500 reports across both centers), report per-region and per-status precision/recall, and quantify the fraction of excluded examinations along with the reasons.","section":"Section 2.1 and Section 3"},{"comment":"The empirical comparisons are reported as single numbers without error bars, multiple seeds, or statistical significance tests. This is concerning because several margins are small (e.g., PolypDiag F1: 97.3 vs 96.3 for EndoFM-LV; CVC-12k Dice: 86.4 vs 85.8 for DINOv3†) and for LIMUC EndoVLM actually trails GastroNet-5M (94.5/74.4 vs 95.5/–). The ablation in Table 2 also appears to have ambiguous checkmark placement: the third row seems to combine Lglo and LMAE, but the rendering is unclear. The paper should report mean and standard deviation over at least three random seeds, use paired significance tests against the strongest baselines, and clarify the exact configuration of each ablation row.","section":"Section 3, Tables 1 and 2"},{"comment":"The zero-shot evaluation is under-specified. The text gives only the prompt 'This is an image of {cls}.' and refers to Fig. 2, but does not state the dataset split, the class vocabulary, the number of trials, or how video-level predictions are formed for PolypDiag (e.g., whether per-frame scores are mean-pooled). The near-perfect AUC values for upper-GI anatomy recognition and the 18% improvement over BiomedCLIP on video disease diagnosis cannot be verified without this protocol. Please provide the full zero-shot setup, per-class AUC/F1, and a confusion matrix; the acknowledged lower-ileum limitation should also be quantified with a per-class result.","section":"Section 3, zero-shot evaluation"},{"comment":"The fine-grained alignment loss has a self-referential structure: the visual representation z_u in Eq. (1) is constructed by attending to frames selected with the text query t_i,j, and Eq. (5) then pulls z_u toward that same text embedding. The soft target in Eq. (4) is also a function of text-text similarities. In principle, this could allow the model to minimize L_fg by matching text embeddings to themselves without acquiring new visual information. The ablation in Table 2 shows that adding L_fg improves downstream segmentation and grading, suggesting that the representation is not trivial; nevertheless, the paper should provide direct evidence that z_u contains visual information beyond the text selector, for example by probing the learned visual features on a frame-level anatomy classification task with ground-truth labels.","section":"Section 2.3, Eqs. (1)-(5)"}],"minor_comments":[{"comment":"The 17 GI regions are referenced as 'aforementioned' but the mapping is only implicit from the Introduction; please list the eight gastroscopy and nine colonoscopy regions explicitly at the point of use.","section":"Section 2.1"},{"comment":"The checkmarks do not visually align with the column headers in the current rendering; please use explicit checked/unchecked cells or a legend.","section":"Table 2"},{"comment":"In the sentence 'boosting AUC by 8.9% and Dice by 19.4%. which highlights...', the period before 'which' should be a comma.","section":"Ablation paragraph"},{"comment":"The repeated use of 'EndoFM-LV' (Wang et al., 2025b) versus 'EndoFM' (Wang et al., 2023) may confuse readers; please define the abbreviations on first use and make the distinction in the comparison table.","section":"Section 3"},{"comment":"The hard target y=1 for normal-normal pairs of the same anatomy is a design choice that differs from the soft-text-similarity used when either item is abnormal; please justify this asymmetry in the text.","section":"Eq. (4)"},{"comment":"The zero-shot figure (Fig. 2) should include axis labels, units, and error bars (if any) to be self-contained, since the main text does not provide the numeric values.","section":"Section 3"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The main risk to the paper is the unvalidated LLM parsing step; this is not an issue of novelty but of evidence. The paper is likely in scope and the proposed method is interesting. If the authors can provide a human audit of the parsing and more rigorous statistics, the contribution could be suitable. Please note that the code link is included but its availability at review time is unknown."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about EndoVLM. First, it is a real engineering contribution: a vision-language pretraining framework that handles the messy reality of GI endoscopy, where one report covers dozens of unlabeled frames. The anatomy-guided sparse pooling (AGSP) is a sensible fix for redundant image-sets, the taxonomy-aware soft targets (PSAA) are a thoughtful way to use clinical structure, and the semantic-concentrated MAE adds a plausible low-level complement. The ablation is honest and each component earns its place: global alignment alone gets 85.2 AUC / 66.2 Dice on the two main tasks, adding fine-grained alignment jumps to 94.1/85.6, and the MAE gives a further small bump. Those numbers are coherent and the evaluation uses external public benchmarks, which is exactly the right way to make a claim like this.\n\nSecond, the soft spot is real and it sits at the center of the method. The Qwen3-based parsing of clinical reports into semantic triplets (finding, anatomy, pathology) has zero reported validation. No clinician agreement, no accuracy statistic, no failure analysis. Because AGSP uses those anatomy labels as queries and PSAA uses the pathology labels to build soft targets, every parsing error propagates directly into the pretraining signal. The same parser also decides which cases are 'ineligible' and filters them out, so systematic parser bias could silently shape the 348K corpus. This is not a minor detail; it is the load-bearing link between the raw data and the claimed representations. The stress-test note is right to flag it.\n\nSome smaller issues: there are no error bars or significance tests, so the differences between EndoVLM and the closest baselines (e.g., 94.5 vs 94.0 AUC on LIMUC) may be noise. The zero-shot protocol is under-specified - prompt details, class selection, and the reported ~100% AUC in upper-GI anatomy all need more scrutiny. The fine-grained alignment has a mildly self-referential flavor: Eq. 1 pools frames by text-query attention, then Eq. 4 pulls those pooled vectors toward the same text embeddings. Global alignment and the MAE probably keep this from collapsing, but it deserves a comment or an experiment. Data and weights are not released, though the code link is a start.\n\nOverall, the central claim is plausible and the paper is worth engaging with seriously. It is not a paradigm shift, but it gives the endoscopy community a reusable pretraining framework and a competitive backbone. Who is this for? Anyone building endoscopy foundation models or doing transfer learning on GI video/image tasks. My recommendation: send it to peer review, but make the Qwen3 parser audit (including clinician validation on a sample) a hard requirement, along with error bars or repeated-run statistics. Without that, the unmeasured label noise makes the exact numbers hard to trust.","headline":"EndoVLM is a genuinely useful endoscopy vision-language pretraining recipe with a convincing ablation, but its unvalidated LLM report-parsing step is a load-bearing risk that a serious referee should require auditing before the performance claims are trusted.","tokens_in":875,"tokens_out":854,"would_cite":true,"duration_ms":22447,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that EndoVLM, pre-trained on 348K endoscopic image-set/report pairs without per-frame labels, outperforms existing general and endoscopy-specific foundation models across downstream tasks and supports zero-shot anatomy…","keywords":["endoscopy","vision-language pre-training","foundation model","image-set report pairing","anatomy-guided sparse pooling","semantic-aware alignment","masked autoencoder","zero-shot anatomy recognition"],"falsifier":"Take a random sample of the 348,000 reports, have expert gastroenterologists label each sentence's anatomical region and normal/abnormal status, and compare against the automatic parser. If agreement is low, especially on abnormal findings, the AGSP queries and PSAA soft targets are unreliable and the reported gains could be an artifact; alternatively, permuting the anatomy labels during pretraining should destroy the PSAA advantage if the alignment is genuinely anatomy-guided.","tokens_in":9747,"feed_emoji":"🩺","tokens_out":13220,"duration_ms":98672,"temperature":0.7,"pith_summary":"EndoVLM tries to establish that routine endoscopic image-set/report pairs, rather than per-frame annotations, are enough supervision to build a reusable endoscopy foundation model. The paper pre-trains on 348K examinations, using each report's anatomical findings as queries that sparsely select the few relevant frames among dozens of redundant images, then aligns the pooled visual features to report semantics through anatomy- and pathology-aware soft targets, and adds masked reconstruction on those selected frames. If the claims hold, a single backbone can handle video polyp diagnosis, polyp segmentation, anatomy recognition, and zero-shot disease classification with simple downstream heads, outperforming existing foundation models and staying competitive with task-specific methods on several benchmarks. The practical significance is that endoscopy generates abundant unannotated image-report data at scale, so the method could turn routine clinical records into training signal.","feed_headline":"348K reports train an endoscopy AI that outperforms other models","feed_subtitle":"Routine image-report pairs, not per-frame labels, power features that transfer zero-shot to new exams.","key_machinery":"The machinery is a jointly trained three-part objective. First, Anatomy-Guided Sparse Pooling computes dot-product relevance between each text-encoded anatomical sentence embedding and every frame embedding, keeps the top K=3 frames per query, and re-normalizes attention over those frames to produce one visual vector per anatomy. Second, Progressive Semantic-Aware Alignment combines a symmetric InfoNCE loss between the image-set average and the whole-report embedding with a fine-grained contrastive loss over the pooled anatomy vectors, using soft targets that depend on matching anatomy and pathological status. Third, Semantic-Concentrated Masked Autoencoder masks 75% of patches in the union of AGSP-selected frames and reconstructs pixels by mean squared error. The text embeddings do double duty as queries for sparse pooling and as anchors for soft-target alignment; the taxonomy-derived soft targets are the piece that injects clinical structure directly into the contrastive distribution.","core_discovery":"The paper's central claim is that its three components close the modality gap between structured clinical reports and unordered, redundant image-sets. Anatomy-Guided Sparse Pooling treats each parsed sentence, a morphological finding tied to one of 17 gastrointestinal regions and a normal/abnormal status, as a query and uses top-K sparse attention to aggregate only the semantically relevant frames into an anatomy-specific visual vector. Progressive Semantic-Aware Alignment first runs patient-level global contrastive alignment, then a fine-grained contrastive loss whose soft targets encode the clinical taxonomy: identical healthy anatomies are not repelled, while abnormal findings within the same anatomy are softly pulled together by textual similarity. Semantic-Concentrated Masked Autoencoder applies pixel reconstruction only to the selected semantic-rich frames, adding low-level geometric detail back into the high-level semantic representation. The paper reports that this combination outperforms general and endoscopy-specific foundation models across downstream tasks, provides near-perfect zero-shot upper-GI anatomy recognition, and stays competitive with task-specific segmentation models using only a minimal head.","pith_inferences":["Editorial inference: if the parser's accuracy on abnormal findings is high, the same image-set/report recipe should transfer to other multi-region imaging modalities with structured reports, such as dermoscopy body mapping or pathology whole-slide reports, but the fixed regional taxonomy would need to be rebuilt.","Editorial inference: the lower-ileum zero-shot weakness the paper attributes to procedural language suggests that reports describing the procedure rather than the mucosa need separate handling or a parser that distinguishes procedural sentences from finding sentences.","Editorial inference: because AGSP discards temporal order and most frames, adding a temporal-context branch over the retained frames could further improve video-level diagnosis; this is not tested in the paper.","Editorial inference: since the public benchmarks are disjoint from the private pre-training corpus, an external multi-center cohort study would be the natural next test of whether the generalization claim holds outside the two source centers."],"forward_implications":["Endoscopy foundation models can be pre-trained from routine clinical reports and unordered image packages, without per-frame labels, because AGSP selects the few salient frames for each anatomical query.","A single pretrained backbone covers both image-level tasks such as polyp diagnosis and anatomy recognition and dense prediction such as polyp segmentation, with a linear head or a single convolutional layer.","In the ablation, adding the taxonomy-aware fine-grained alignment to global alignment raises CVC-12k Dice by 19.4 points and LIMUC AUC by 8.9 points, so the anatomy/pathology soft targets, not the reconstruction loss, carry most of the gain.","Zero-shot transfer works for anatomy recognition and video disease diagnosis through the aligned visual-semantic space, meaning the model can be queried with text prompts instead of being fine-tuned."],"supporting_citations":[{"why":"Supplies the large language model that parses each report into finding-anatomy-pathological-status triplets, the input on which AGSP and PSAA depend.","marker":"(Yang et al., 2025)"},{"why":"Defines the global image-text contrastive alignment paradigm and serves as a general vision-language baseline the paper must beat.","marker":"(Radford et al., 2021)"},{"why":"Provides the InfoNCE loss used for both the global cross-patient alignment and the fine-grained alignment distributions.","marker":"(Oord et al., 2018)"},{"why":"Supplies the masked-autoencoder paradigm that SC-MAE adapts to semantic-rich frames and is itself a general baseline.","marker":"(He et al., 2022)"},{"why":"Provides DINOv2, a strong self-supervised visual baseline compared across the downstream tables.","marker":"(Oquab et al., 2023)"},{"why":"Sets the endoscopy video foundation-model baseline and the evaluation protocol followed for polyp diagnosis and CVC-12k segmentation.","marker":"(Wang et al., 2025b)"},{"why":"Supplies the Hyper-Kvasir dataset used for zero-shot anatomy recognition across the upper and lower GI tracts.","marker":"(Borgli et al., 2020)"},{"why":"Provides the LIMUC ulcerative-colitis severity task and its evaluation setting for the severity-grading comparison.","marker":"(Polat et al., 2023)"},{"why":"Defines the polyp-segmentation generalization protocol and a strong task-specific baseline for comparison.","marker":"(Dong et al., 2021)"}],"fun_headline_variants":["Endoscopy AI trained on 348K report-image pairs beats other models","Anatomy-guided alignment lets endoscopy AI learn from clinical reports","Zero-shot endoscopy AI from paired clinical reports and images","EndoVLM: sparse attention to clinical text improves endoscopy vision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pretraining signal depends on the language model parsing each clinical report into correct finding-anatomy-status triplets and on the filtering step not discarding a biased subset of cases, and the paper reports no human-validation statistic for that parsing.","fun_headline_variants_meta":{"raw":{"variants":["Endoscopy AI trained on 348K report-image pairs beats other models","Anatomy-guided alignment lets endoscopy AI learn from clinical reports","Zero-shot endoscopy AI from paired clinical reports and images","EndoVLM: sparse attention to clinical text improves endoscopy vision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000815,"raw_usage":{"total_tokens":3603,"prompt_tokens":1009,"completion_tokens":2594,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":2520}},"tokens_in":625,"tokens_out":2594,"duration_ms":17612,"temperature":1.0,"reasoning_tokens":2520,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:39:27.121372+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the 348,000 reports, have expert gastroenterologists label each sentence's anatomical region and normal/abnormal status, and compare against the automatic parser. If agreement is low, especially on abnormal findings, the AGSP queries and PSAA soft targets are unreliable and the reported gains could be an artifact; alternatively, permuting the anatomy labels during pretraining should destroy the PSAA advantage if the alignment is genuinely anatomy-guided.","supporting_citations":[],"review_version":1}