{"id":"d7aad88b-5ef4-40cb-8876-bbcc02cbf21d","arxiv_id":"2507.17323","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Multi-view scene representations with hash-based retrieval enable interpretable colorectal polyp diagnosis, beating single-image baselines on the authors' new PolypScene-250 and PolypScene-80 datasets.","lead":"EndoFinder is a retrieval system that combines multiple endoscopic views of a polyp into a single latent scene code, then finds the most similar historical cases to support diagnosis. Instead of a black-box classification, it shows clinicians confirmed past cases, and the authors report improved re-identification and pathology classification over existing methods on their new PolypScene datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Data split between scene-encoder training (PolypScene-2k) and evaluation (PolypScene-250/80) is never stated; overlap would inflate re-identification and classification results.","rationale":"The claim that EndoFinder-S outperforms all other methods rests on the implicit assumption that the training pool for the scene encoder is disjoint from the evaluation set. The paper's own description in Section 4 says the Scene Representation Transformer is trained on PolypScene-2k (2,219 polyps) and evaluated on PolypScene-250 (250 polyps), but it never states that these sets do not intersect. Given that both are multi-view polyp collections and the names suggest a hierarchy, overlap is plausible. Table 1 does not resolve this because it reports counts and purposes, not split membership. PolypScene-80 being a subset of PolypScene-250 extends the issue to the clinical classification experiment. Because the task is explicitly to retrieve the same polyp across viewpoints, any overlap between training and test scenes subverts re-identification: the model can match by memorized identity rather than generalized appearance. The comparisons to DINO-S, MAE-S, and to fine-tuned classifiers are then not apples-to-apples unless those methods also had access to the same unlabeled target scenes. This is the weakest load-bearing point; it precedes other concerns such as small sample size and k selection. The reader identified the same issue. A metadata check can settle it. If the sets are disjoint, the paper's central conclusion survives this audit. If they are not, the quantitative headline results must be recomputed on a clean split. Therefore the verdict remains CONDITIONAL; no adjustment beyond the reader's recommendation is needed.","tokens_in":15080,"tokens_out":6115,"duration_ms":61461,"concrete_test":"Using the released metadata or authors' confirmation, compute the exact intersection of unique polyp/scene identifiers among PolypScene-2k, PolypScene-250, and PolypScene-80. If identifiers are not available, hash frames and run near-duplicate image matching across the training and evaluation sets. If the intersection is empty, the split concern is resolved. If any scene overlaps, retrain the scene encoder on PolypScene-2k excluding all overlapping scenes and rerun Tables 4 and 5; a material drop in µAP/Acc@1 or in classification metrics, to the point where EndoFinder-S no longer beats EndoFinder-I and DINO-S, would show the reported advantage is leakage-driven.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central result that EndoFinder-S 'outperforms all other methods' depends on an unstated data split: the 2,219 scenes in PolypScene-2k used to train the Scene Representation Transformer (Section 3.4, Algorithm 1, losses in Eqs. 8-11) must be disjoint from the 250 scenes in PolypScene-250 used for re-identification (Section 4, Table 4). The paper nowhere states this, and Table 1's dataset list does not encode training/evaluation partition membership. If PolypScene-250 is a subset of, or overlaps with, PolypScene-2k, then the exact query/reference polyp identities were seen during contrastive and reconstruction pretraining; the re-identification task becomes recognition of training identities, not retrieval of unseen lesions. Because PolypScene-80 is explicitly a subset of PolypScene-250 (Section 4), the clinical classification result inherits the contamination, and the fine-tuned baselines in Table 5, which were not pretrained on the target scenes, are not fairly comparable. Section 6's limitations mention segmentation-mask dependence and view-number requirements but not this split.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes EndoFinder, a retrieval-based framework for explainable colorectal polyp diagnosis. The method first trains a ViT-based image encoder using contrastive learning and masked reconstruction guided by polyp segmentation masks, then a Scene Representation Transformer that fuses multiple endoscopic views of a polyp into a latent scene representation. This representation is binarized into hash codes to enable efficient Hamming-distance retrieval from a database of historical cases, and the final diagnosis is obtained by a k-NN majority vote. The authors introduce the PolypScene-250 dataset (with the PolypScene-80 pathology-labeled subset), report re-identification and classification experiments, and claim that EndoFinder outperforms existing methods while offering interpretable diagnostic references.","tokens_in":15336,"tokens_out":5058,"duration_ms":47719,"significance":"The paper addresses a clinically relevant problem and offers a plausible direction for interpretable AI in colonoscopy. The release of a new multi-view polyp dataset and the source code are valuable contributions, and the idea of modeling a polyp as a 2.5D/3D scene rather than a single image is a reasonable and well-motivated extension. If the empirical results are robust, the work would show a meaningful advance for retrieval-based polyp diagnosis. However, the central 'outperforms' claims currently depend on an unstated data-split assumption and on comparisons lacking statistical justification, so the significance is not yet fully established.","major_comments":[{"comment":"The manuscript never states whether the 250 polyps in PolypScene-250, and its subset PolypScene-80, are disjoint from the 2,219 polyps in PolypScene-2k used to train the Scene Representation Transformer (Section 3.4, Algorithm 1). Because PolypScene-2k is described as containing 2,219 polyps and PolypScene-250 as containing 250, the evaluation sets could be subsets of the training set. If so, the re-identification results in Table 4 (µAP 0.71, Acc@1 0.74) and the classification results in Table 5 (AUC 85.59, ACC 78.75, F1 81.00) would be inflated by the model having seen the exact polyp identities during training, and the comparison to fine-tuned classifiers that were not pretrained on these scenes would be unfair. Please state explicitly whether the splits are disjoint, and if they are not, re-run the evaluation on held-out polyps or provide a clear justification for why the overlap does not affect the conclusions.","section":"Section 4 / Table 1"},{"comment":"The classification experiment on PolypScene-80 uses five-fold cross-validation on just 80 polyps, yet no error bars, confidence intervals, or significance tests are reported. With roughly 16 test samples per fold, the observed improvement of EndoFinder-S (AUC 85.59) over the best fine-tuned baseline ViT-L (AUC 81.74) may not be statistically reliable. In addition, the k-NN parameter k=6 is selected without sensitivity analysis. Please report per-fold results with standard deviations, conduct a significance test (e.g., paired bootstrap or Wilcoxon), and include an ablation over k.","section":"Section 5.2 / Table 5"},{"comment":"The hash length K, which is a key hyperparameter of the binarization in Section 3.5, is never stated in the experimental section, although the speed-accuracy trade-off reported in Figure 5 depends on it. The paper also reports only a relative '4x speed improvement' without absolute query times. Please specify K and provide absolute timings (e.g., milliseconds per query) for both the float and binary retrieval cases to support the real-time claim.","section":"Section 5.3 / Figure 5"},{"comment":"The 'entropy' regularization term is defined as the negative logarithm of the minimum distance to any negative sample. This is not the standard entropy regularizer, the logarithm becomes negative when the minimum distance exceeds 1, and the gradient is singular as the minimum distance approaches zero. The paper provides no motivation or ablation for this choice, and the naming is misleading. Please clarify the intended behavior and either provide a theoretical or empirical justification or replace the term with a well-understood uniformity loss.","section":"Section 3.3, Eq. (4) and Section 3.4, Eq. (9)"}],"minor_comments":[{"comment":"The checkmark symbols used in Table 3 are not defined in the caption; please add an explicit statement such as '√ indicates that the corresponding view is used.'","section":"Table 3"},{"comment":"The sentence 'Each polyp in the PolypScence dataset has four views, which are randomly labeled as Q1, Q2, R1, and R2' should specify the random seed or the exact partition, because the re-identification results depend on which views are assigned to the query and reference groups.","section":"Section 5.1.2"},{"comment":"The limitations paragraph discusses dependence on segmentation masks and the need for multiple views, but it does not mention the possibility of training/evaluation overlap; once the data-split issue is resolved, this should be addressed explicitly.","section":"Section 6"},{"comment":"The axis labels are not clearly readable in the figure; please add explicit axis titles and units.","section":"Figure 5"},{"comment":"A few references are cited in the text but appear incomplete or inconsistently formatted (e.g., the citation to 'Brearley et al., 2022' in the Introduction); please proofread the reference list.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the scope of the journal, but the central empirical claims are currently contingent on the authors' ability to demonstrate that the evaluation sets are disjoint from the scene-encoder training set. I recommend requesting the authors to clarify the dataset partition, provide per-fold statistics and significance testing, and report the hash length and absolute timing. If these concerns are addressed convincingly, the paper could be suitable for publication; without them, the comparative claims are not verifiable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nThe thing to know about this paper: it's a useful, well-put-together retrieval framework for polyp diagnosis, but the main empirical claim rests on a data split that is never stated. If the PolypScene-250/80 evaluation scenes are a subset of the PolypScene-2k training set for the scene encoder, the re-identification and classification numbers are inflated by memorization. The paper shows no statement that they are disjoint, and the limitations section doesn't mention it. That's a load-bearing omission, not a cosmetic one.\n\nWhat's genuinely new: treating a polyp as a multi-view scene and learning a latent scene representation with contrastive plus reconstruction losses, then binarizing to hash codes for fast retrieval. The dataset contribution, PolypScene-250 with the 80-case histopathology subset, is a real asset. The experiments are reasonably designed—baselines include DINO, SSCD, MAE, and the diagnostic comparison against fine-tuned classifiers is a fair setup. The method's logic is clear and the code is promised.\n\nSoft spots, in order of severity:\n\n1. The split issue above. This has to be clarified before the central claims can be trusted.\n2. The pathology classification uses 80 cases with five-fold CV but no error bars. The gap between EndoFinder-S (85.59 AUC) and the best baseline (ViT-L 81.74) is 3.85 points—on 80 samples that could be noise. The paper needs confidence intervals or per-fold results.\n3. Hyperparameters k=6 and hash length K are chosen without sensitivity analysis. A retrieval framework that's robust to k would be more credible.\n4. Minor: the paper says \"outperforms all other methods\" based on a single split of data that is itself a subset of the test set; the comparisons to fine-tuned baselines are not apple-to-apple if the baselines never saw the scene-level training data.\n\nMy take: the idea is solid, the execution is decent, and the dataset could be valuable. But the missing split statement is exactly the kind of thing that gets a paper rejected after review. I'd send it to peer review—the reviewers can force the authors to disclose and re-run if needed—but I wouldn't cite the numbers until that's resolved.\n\nRecommendation: accept for review with a request for split disclosure, error bars, and a k sensitivity study.","headline":"Solid retrieval framework with a valuable dataset, but the unstated train/test split undercuts the headline numbers.","tokens_in":15858,"tokens_out":2283,"would_cite":false,"duration_ms":22612,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Treating a polyp as a three-dimensional scene rather than a single image lets a retrieval system identify the polyp across views and infer its pathology from similar past cases.","keywords":["Polyp diagnosis","Scene representation","Polyp retrieval","Semantic hashing","Colorectal cancer","Colonoscopy","Contrastive learning","Explainable AI"],"falsifier":"A decisive check is to compare polyp identities across the three PolypScene datasets: if any training polyp from PolypScene-2k also appears in PolypScene-250 or PolypScene-80, the claim as stated is unsupported. The cleaner experiment is to retrain the scene encoder on a polyp-disjoint subset and evaluate on held-out polyps, or to test on an external multi-view polyp dataset; the re-identification and pathology numbers should stay close to the reported µAP 0.71 and AUC 85.59 if the result is real generalization.","tokens_in":14920,"feed_emoji":"🔍","tokens_out":10306,"duration_ms":100460,"temperature":0.7,"pith_summary":"During a colonoscopy, a doctor must often decide whether a polyp is harmless or precancerous before the pathology report exists. The paper argues that the right way to help is not a black-box classifier but a retrieval system: treat each polyp as a three-dimensional scene, fuse the several endoscopic views of it into one latent representation, then fetch the most similar historical polyps with known diagnoses. EndoFinder, built this way, outperforms single-image retrieval and fine-tuned supervised classifiers on the paper's datasets, reaching µAP 0.71 and Acc@1 0.74 on scene-based re-identification and AUC 85.59, ACC 78.75, F1 81.00 on pathology classification. The diagnosis is a majority vote of the retrieved neighbors, so the clinician can look at the supporting cases. Hashing the scene representations to binary codes makes the search fast enough for real-time use, and adding or deleting cases from the database needs no retraining.","feed_headline":"Multi-view scene retrieval beats single-image AI for polyp diagnosis","feed_subtitle":"Retrieval shows doctors similar past cases behind each diagnosis instead of a black-box score.","key_machinery":"The load-bearing component is the Scene Representation Transformer: a one-layer Transformer that takes the [CLS] tokens produced by a frozen ViT-L/16 image encoder for each of four endoscopic views and fuses them, via self-attention and a learnable scene token, into one latent scene vector $S$. The scene encoder is trained with a scene-level InfoNCE loss, an entropy regularizer that pushes representations apart, and a masked-reconstruction objective in which the remaining views reconstruct a masked view through cross-attention, a pose-free mechanism. At inference, $S$ is sign-binarized into a hash code in $\\{-1,1\\}^K$, and a ball tree over Hamming distance retrieves the nearest historical cases. This design carries the argument because every re-identification and pathology result is a property of this fused scene space, not of any single-view feature.","core_discovery":"The central claim is that polyps with similar latent scene representations share clinical characteristics, and that a retrieval-based system built on that assumption can match or beat conventional supervised classifiers while staying interpretable. The evidence is EndoFinder-S, the variant that fuses four views with a Scene Representation Transformer: on PolypScene-250 it reaches µAP 0.71 and Acc@1 0.74 for re-identifying the same polyp across views, against 0.68/0.70 for simple feature averaging, and on PolypScene-80 it reaches AUC 85.59, ACC 78.75, and F1 81.00 for pathology classification, above the best fine-tuned supervised model (ViT-L at 81.74/78.75/80.08). The same representation, binarized into hash codes and searched through a ball tree with Hamming distance, gives a 4x retrieval speedup with a 6.7% µAP drop. The authors interpret this as evidence that moving from single images to multi-view scene representations is the decisive step: the model does not learn a direct image-to-label mapping, it learns an embedding space in which historical cases with known outcomes are the explanation.","pith_inferences":["Editorial inference: the cleanest test of the 'similar scene, similar label' hypothesis would be an external multi-view polyp dataset with pathology labels, since the three PolypScene datasets described in the paper come from the same data family.","Editorial inference: a deployment-oriented follow-up should measure how performance degrades when only one or two views can be captured during a real procedure; the paper's ablations vary view count within the same four-view dataset.","Editorial inference: the privacy argument for retrieval applies fully only to the database stage; deleting a row removes that case from retrieval, but the image encoder was pretrained on all of Polyp-18k, so information from those images remains in the model weights.","Editorial inference: the framework could be evaluated as a clinician-in-the-loop decision aid, asking whether access to retrieved reference images changes a doctor's management decision or time-to-decision, not just the model's AUC."],"forward_implications":["During colonoscopy, the system can show the clinician the most similar already-diagnosed polyps behind each predicted label, making the AI output checkable against real cases.","Diagnostic knowledge is stored in a searchable database: adding a newly confirmed polyp is an insert, and deleting a patient's case removes its retrieval influence, with no model retraining.","Recording several views of a polyp becomes clinically useful, since multi-view scene fusion is what lifts re-identification and classification above single-view features.","Hash-based search gives a 4x speedup over cosine similarity on float features with only a small drop in µAP, which is the evidence that real-time use is feasible.","The paper's authors propose extending the same scene-retrieval paradigm to other lesion types and to additional inputs such as patient history in future work."],"supporting_citations":[{"why":"Earlier EndoFinder paper whose polyp-aware image encoder and 2D retrieval pipeline this work extends to multi-view scene retrieval.","marker":"Yang et al., 2024a"},{"why":"Provides the unposed scene representation and the cross-attention masked reconstruction objective used to train the scene encoder.","marker":"Sajjadi et al., 2023"},{"why":"Introduces the set-latent scene representation that motivates the Scene Representation Transformer.","marker":"Sajjadi et al., 2022"},{"why":"Supplies the augmentation strategy and InfoNCE-plus-entropy contrastive losses used for both the image and scene encoders.","marker":"Pizzi et al., 2022"},{"why":"Source for deep Cauchy hashing and Hamming-space retrieval that motivates the hash-code stage.","marker":"Cao et al., 2018"},{"why":"Supplies the ball-tree index used to accelerate Hamming-distance nearest-neighbor search.","marker":"Brearley et al., 2022"},{"why":"Supports the kNN-on-embeddings classification paradigm that underpins retrieval-based diagnosis.","marker":"Doerrich et al., 2024"}],"fun_headline_variants":["Multi-view polyps beat single-image AI for diagnosis","Retrieval system makes polyp AI explainable","Scene-based retrieval improves polyp classification","EndoFinder: interpretable polyp diagnosis via retrieval","Polyps as scenes: AI retrieves similar cases"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that no polyp used to train the scene encoder on PolypScene-2k appears in the PolypScene-250 or PolypScene-80 evaluation sets; the paper never states this training/evaluation split, and if the sets overlap, the reported gains over baselines would be inflated by memorization rather than generalization.","fun_headline_variants_meta":{"raw":{"variants":["Multi-view polyps beat single-image AI for diagnosis","Retrieval system makes polyp AI explainable","Scene-based retrieval improves polyp classification","EndoFinder: interpretable polyp diagnosis via retrieval","Polyps as scenes: AI retrieves similar cases"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00037,"raw_usage":{"total_tokens":2044,"prompt_tokens":1069,"completion_tokens":975,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":685,"completion_tokens_details":{"reasoning_tokens":904}},"tokens_in":685,"tokens_out":975,"duration_ms":8417,"temperature":1.0,"reasoning_tokens":904,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:51:08.135739+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check is to compare polyp identities across the three PolypScene datasets: if any training polyp from PolypScene-2k also appears in PolypScene-250 or PolypScene-80, the claim as stated is unsupported. The cleaner experiment is to retrain the scene encoder on a polyp-disjoint subset and evaluate on held-out polyps, or to test on an external multi-view polyp dataset; the re-identification and pathology numbers should stay close to the reported µAP 0.71 and AUC 85.59 if the result is real generalization.","supporting_citations":[{"cited_title":", author Mahendran, A","cited_arxiv_id":null,"evidence_quote":"Provides the unposed scene representation and the cross-attention masked reconstruction objective used to train the scene encoder."},{"cited_title":", author Meyer, H","cited_arxiv_id":null,"evidence_quote":"Introduces the set-latent scene representation that motivates the Scene Representation Transformer."},{"cited_title":", author Roy, S.D","cited_arxiv_id":null,"evidence_quote":"Supplies the augmentation strategy and InfoNCE-plus-entropy contrastive losses used for both the image and scene encoders."},{"cited_title":", author Long, M","cited_arxiv_id":null,"evidence_quote":"Source for deep Cauchy hashing and Hamming-space retrieval that motivates the hash-code stage."},{"cited_title":", author Bose, K.R","cited_arxiv_id":null,"evidence_quote":"Supplies the ball-tree index used to accelerate Hamming-distance nearest-neighbor search."},{"cited_title":", author Archut, T","cited_arxiv_id":null,"evidence_quote":"Supports the kNN-on-embeddings classification paradigm that underpins retrieval-based diagnosis."}],"review_version":1}