{"id":"7e17ae72-9fea-45f7-b764-98d668788d87","arxiv_id":"2412.19218","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A DETR-based model with ResNet50 detects and classifies bleeding tissue in capsule endoscopy frames, reporting 98.28% accuracy on a self-made validation split.","lead":"This paper applies a transformer-based object detector called DETR to automatically find and classify bleeding versus non-bleeding frames in wireless capsule endoscopy videos. The authors report high classification accuracy on a validation split and a third-place finish in the Auto-WCEBleedGen challenge, but they do not report quantitative results on the independent test sets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The quantitative headline rests on a self-created validation split; official test sets appear only as images, so the claimed accuracy and mAP may not transfer to unseen WCE data.","rationale":"I agree with the reader's weakest_assumption: the load-bearing premise is that results on the authors' self-created 80:20 validation split transfer to the independent official test sets. The paper is transparent that Table 1 reports validation-set numbers, and the only test-set evidence is illustrative sample images plus Grad-CAM visualizations. The '3rd place' statement implies hidden leaderboard metrics, but those are not disclosed, so the reader cannot verify that the central claim holds on the challenge's evaluation. This is not an internal mathematical inconsistency; it is an unmet evidential requirement for the clinical-generalization language used in the introduction and discussion. The reader's verdict is already CONDITIONAL, and my stress-test does not move it: the condition should be made explicit, namely that the authors must report quantitative metrics on Test Set 1 and Test Set 2 using the official evaluation script and provide a complete enough code/weight snapshot to reproduce those numbers. I considered whether the ambiguous classification head or the lack of baselines might be equally load-bearing, but those affect reproducibility and context rather than the truth of the headline claim; the test-set gap is the single concern that, if it lands, would invalidate the paper's practical conclusion.","tokens_in":4070,"tokens_out":3096,"duration_ms":32830,"concrete_test":"Obtain the official Auto-WCEBleedGen Test Set 1 and Test Set 2 (Zenodo DOI 10.5281/zenodo.10642779), run the public GitHub implementation with the listed 500-epoch fine-tuning configuration, and compute Accuracy, Recall, F1, AP@50, mAP, and detection Recall using the challenge's official evaluation protocol. Compare these numbers directly with Table 1. If any metric drops by more than two or three points (e.g., mAP < 0.65 or F1 < 95%), the validation-only report is not representative of test-set performance; if all metrics stay within a small margin, the generalization concern is resolved and the 3rd-place claim becomes verifiable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claims in Table 1 are computed on an 80:20 split of the challenge training set (Section 2.1), with both classes equally represented. The official Test Set 1 and Test Set 2 are described as independently collected WCE frames from more than 30 patients (Section 2.1), but Section 3 reports only visual samples from these sets (Figures 4 and 5) and no quantitative metrics. The abstract's statement 'This earned us a 3rd place position in the challenge' implies that official evaluation was performed, yet no official scores are reported or compared with Table 1. Because the validation split shares acquisition characteristics with the training set, it measures intra-dataset fit rather than generalization to new patients or acquisition conditions. If quantitative test-set metrics are materially lower than the validation metrics, the paper's stated clinical motivation—automating analysis of unseen WCE frames—is not supported by the evidence presented. The manuscript itself acknowledges using the official test data in Section 6, making the omission of those numbers a resolvable but currently load-bearing gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes a DETR-based end-to-end model for bleeding and non-bleeding frame classification and bleeding-region detection in wireless capsule endoscopy (WCE) images, developed for the Auto-WCEBleedGen Version 1 challenge. The model uses a ResNet50 backbone, a transformer encoder-decoder for detection, and an appended feedforward layer for three-class classification (bleed, non-bleed, background). It is fine-tuned for 500 epochs on an 80:20 split of the challenge's 2618-frame training set. The authors report classification accuracy 98.28%, recall 96.79%, F1 98.37%, and detection AP@50 0.7447 / mAP 0.7328 on their self-created validation split, and state that this earned third place in the challenge. They also show qualitative detection and Grad-CAM examples from validation set and from the two official test sets, but no quantitative test-set metrics are reported.","tokens_in":4197,"tokens_out":1882,"duration_ms":19755,"significance":"If the reported validation-set numbers were confirmed on the independent official test sets, the paper would provide a useful demonstration that a single DETR-based model can jointly perform bleeding-region detection and frame-level classification for WCE, a task with genuine clinical workflow implications. The paper ships its code publicly, which is a concrete asset for reproducibility, and the end-to-end training without hand-designed components (anchors, NMS) is a clean design choice. However, the evidence currently presented does not establish generalization to unseen WCE data: the headline numbers are computed on a split of the challenge training set, while the independently collected test sets appear only as visual examples. The clinical-motivation claim therefore remains unquantified, and the third-place statement is not verifiable from the manuscript. The gap is fixable and the work is a reasonable challenge report, but as a journal submission the missing test-set evaluation is load-bearing.","major_comments":[{"comment":"The central quantitative claims are computed on an 80:20 split of the challenge training set, not on the independently collected test sets. Section 2.1 states that the test dataset contains frames from more than 30 patients, but Section 3 reports only sample detection images and Grad-CAM visualizations from Test Set 1 and Test Set 2, with no quantitative evaluation. The abstract's claim of '3rd place position' implies that official test-set metrics exist, but they are not reported or compared with Table 1. As written, the paper supports only intra-dataset fit, not the stated clinical claim of automating analysis of unseen WCE frames. The authors should add the official test-set metrics (classification accuracy/recall/F1 and detection AP/mAP and recall) and, if available, the challenge leaderboard scores.","section":"Section 2.1 and Section 3 (Table 1, Figures 4–5)"},{"comment":"The reported metrics have no confidence intervals, no number of repeated runs, and no statistical significance assessment. Given that the validation split is balanced and relatively small (roughly 524 frames), a single run with high metrics could be unstable. The paper should report variance across at least a few seeds or provide a bootstrap confidence interval, especially for the detection mAP, which (at 0.7328) is materially lower than the classification F1 and may be sensitive to the exact split.","section":"Section 3 (Table 1)"},{"comment":"The detection results are reported without comparison to any baseline, the challenge's own evaluation protocol, or an ablation of the loss-weight choices (Hungarian 1, cross-entropy 5, IoU 2). Since the loss weights and the frame-level bleeding threshold of 0.5 are free parameters, the paper should at least show that the reported metrics are not strongly dependent on these choices, or justify them by reference to the official evaluation settings.","section":"Section 2.2.2 and Table 1"}],"minor_comments":[{"comment":"The sentence 'we append to, to the output of DETR' contains a duplicated comma and should be rewritten, for example: 'we append a single-layer FFN with 3 neurons to the output of DETR.'","section":"Section 2, paragraph 4"},{"comment":"The phrase 'our model yield itself to parallel processing' is ungrammatical; it should be 'lends itself to parallel processing'.","section":"Section 4, Discussion"},{"comment":"The sentence describing the test dataset — 'an independently collected WCE data containing bleeding and non-bleeding frames of more than 30 patients suffering from acute, chronic, and occult GI bleeding' — would be clearer as 'an independently collected WCE dataset containing' to match the singular 'data set' usage elsewhere.","section":"Section 2.1"},{"comment":"The evaluation-metrics paragraph defines Recall and F1 only implicitly; for completeness, the authors should specify whether classification recall is per-class macro-averaged or balanced, and whether detection recall is computed at a fixed IoU threshold or averaged over 0.5:0.95 as implied by 'Recall (@0.5:0.95)'.","section":"Section 3"},{"comment":"Reference [8] is an Authorea preprint; since the challenge also has an official website, the authors should cite the final published challenge description if one exists, and ensure the dataset references [9] and [10] include the version and access date consistent with the manuscript.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a straightforward challenge-report manuscript. The main concern is that the authors had access to the official test sets (they acknowledge this in Section 6) yet chose to show only qualitative examples, while the quantitative headline rests on a split of the training set. The fix is straightforward (add the official numbers), so I recommend major revision rather than rejection. I would also encourage the editor to check with the challenge organizers whether the reported third-place ranking is consistent with public leaderboard data, as this claim is central to the paper's credibility but is not verifiable from the manuscript itself."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the Auto-WCEBleedGen challenge report from Alawode et al. The bottom line: this is a clean application of DETR to wireless capsule endoscopy bleeding detection, with a three-neuron classifier appended to (or replacing) the head, and the only real numbers come from an 80:20 split of the challenge training set. That split is the problem. The official Test Sets 1 and 2, independently collected from more than 30 patients, appear only as qualitative figures. The abstract says they placed third in the challenge, so those test-set scores exist. Leaving them out makes the headline accuracy/F1/mAP unverifiable for new-patient generalization.\n\nWhat the paper does well: the code is public, the training setup is described in enough detail to reproduce (optimizer, per-module learning rates, loss weights, 500 epochs), and the validation split is balanced. For a challenge report it is reasonably honest, and there is no circular reasoning—the metrics are not defined in terms of fitted constants from the same evaluation. The clinical motivation is legitimate and the figures are easy to follow.\n\nSoft spots, in order. First, the missing official test-set numbers. Section 6 explicitly acknowledges use of the test data, so the authors had access. Without those numbers, the stated clinical efficacy—automating analysis of unseen WCE frames—is not supported by the evidence in this manuscript. Second, there are no baselines or ablations. It is impossible to know whether the DETR adaptation beats a simple ResNet classifier or an earlier CNN detector on this benchmark. Third, the classification head description is ambiguous: DETR outputs 80 classes; the authors say they \"append\" a three-neuron FFN, but they likely replace the classification head. A diagram or one sentence would fix it. Fourth, no confidence intervals or repeated runs. Minor, given the fixed dataset. Fifth, typos (\"append to, to the output\") and the table's recall@0.5:0.95 column is never explained in the text.\n\nThe stress-test note is exactly right: the self-created validation split measures intra-dataset fit, not cross-patient generalization. That is the load-bearing gap.\n\nThis paper is for the WCE challenge community and for anyone who wants a baseline DETR result on Auto-WCEBleedGen. It is not a methodological contribution. Still, it deserves a serious referee: the code is real, the problem is relevant, and the missing test-set metrics are a straightforward, mandatory revision. I would not desk reject it. Ask the authors for official test-set scores, a comparison with at least one non-transformer baseline, and a clarified head description, then accept or reject on those numbers.\n\nRecommend reading group? Maybe, if someone in the group cares about capsule endoscopy AI. I wouldn't cite it in my own work yet.","headline":"A DETR challenge entry for WCE bleeding detection whose real numbers rest on a self-created validation split, with the official test-set scores omitted; the code and setup are solid, but the central generalization claim needs those missing numbers.","tokens_in":4805,"tokens_out":2517,"would_cite":false,"duration_ms":23988,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single end-to-end transformer model detects and classifies bleeding in wireless capsule endoscopy frames, reporting 98.28% validation accuracy.","keywords":["wireless capsule endoscopy","bleeding detection","DETR","transformer","object detection","image classification","Auto-WCEBleedGen","deep learning"],"falsifier":"Run the released model on the official Auto-WCEBleedGen Test Sets 1 and 2 and compute classification accuracy and detection mAP; if these fall materially below 98.28% accuracy or 0.7328 mAP, the claimed transfer to unseen WCE data is refuted.","tokens_in":3801,"feed_emoji":"🩸","tokens_out":9899,"duration_ms":78413,"temperature":0.7,"pith_summary":"This paper attempts to show that a single end-to-end transformer model can handle both tasks needed to automate wireless capsule endoscopy review: classifying each frame as bleeding or non-bleeding, and drawing bounding boxes around bleeding regions. Building on the DETR architecture with a ResNet50 backbone, the authors fine-tune pretrained weights on the Auto-WCEBleedGen challenge training set. On their own 80:20 validation split they report classification accuracy of 98.28%, recall of 96.79%, and F1 of 98.37%, along with detection AP@0.5 of 0.7447 and mAP of 0.7328. The claim is that the same weights perform both detection and classification, removing the need for separate pipelines and hand-designed post-processing such as non-maximum suppression. If those validation numbers transfer to the challenge's independent test sets, the model could be a practical aid for clinicians reviewing thousands of frames per procedure.","feed_headline":"One transformer model flags GI bleeding frames at 98% accuracy","feed_subtitle":"End-to-end model localises bleeding regions and classifies frames; validation F1 is 98.37%.","key_machinery":"The load-bearing mechanism is DETR's set-based transformer architecture: a ResNet50 CNN produces two-dimensional features, a transformer encoder attends over them with positional encodings, and a decoder with a fixed set of learned object queries outputs a fixed-size set of predictions. A bipartite matching loss, combining the Hungarian algorithm with cross-entropy and IoU terms, assigns each prediction to a ground-truth box one-to-one, which removes hand-designed components such as anchor boxes and non-maximum suppression. Transfer learning from published DETR/ResNet50 weights makes fine-tuning feasible on only 2618 frames.","core_discovery":"The central claim is that a fine-tuned DETR (Detection Transformer), with ResNet50 feature extraction and a three-neuron feedforward head for bleed, non-bleed, and background classes, detects and classifies bleeding in capsule endoscopy frames in one forward pass. Using the Hungarian-based bipartite matching loss with weights of 1, 5, and 2 for the Hungarian matching, cross-entropy, and IoU terms, the model is trained end-to-end without anchor boxes or non-maximum suppression. On the Auto-WCEBleedGen v1 validation split it attains 98.28% classification accuracy, 96.79% recall, 98.37% F1, and detection metrics of AP@0.5 = 0.7447, mAP = 0.7328, and recall at 0.5:0.95 = 0.7706. A frame is called bleeding if any detected region has a bleeding probability above 0.5. This placed the entry third in the challenge.","pith_inferences":["The paper reports no quantitative metrics on the official Test Sets 1 and 2, so the strongest claim currently rests on the self-created validation split; generalisation to the independent test sets remains an open question until those numbers are published.","The gap between 98% classification accuracy and 0.73 detection mAP suggests the model may be better suited to triage (flagging bleeding frames) than to precise boundary delineation, pointing to detection-specific heads or more annotated boxes as a likely next improvement.","Because the DETR framework is label-agnostic, the same fine-tuning recipe could be extended to other capsule-endoscopy lesions such as angiodysplasia or erosions by changing the output classes."],"forward_implications":["A single model can perform both frame-level classification and region-level detection in one forward pass, simplifying the WCE analysis pipeline.","Fine-tuning a pretrained DETR on a few thousand medical images yields strong classification metrics, suggesting transfer learning is a viable route for small medical datasets.","The frame-level rule (bleeding if any region exceeds 0.5 probability) provides a simple, clinically interpretable decision boundary.","The reported detection mAP of 0.7328 indicates that while frame triage is accurate, precise localisation of bleeding regions is the harder sub-problem."],"supporting_citations":[{"why":"Defines the DETR architecture and the bipartite matching loss that the paper adapts for end-to-end detection.","marker":"[1]"},{"why":"Supplies the ResNet50 backbone used for feature extraction from WCE frames.","marker":"[2]"},{"why":"Provides the Hungarian assignment algorithm used in the bipartite matching loss.","marker":"[3]"},{"why":"Defines the (generalized) IoU loss component used for bounding box regression.","marker":"[5]"},{"why":"Provides the pretrained DETR/ResNet50 weights that make fine-tuning feasible on the small dataset.","marker":"[6]"},{"why":"Describes the Auto-WCEBleedGen challenge, its evaluation protocol, and the official competition context.","marker":"[8]"},{"why":"Is the training dataset of bleeding and non-bleeding frames with bounding boxes.","marker":"[9]"},{"why":"Is the independent test dataset used for the qualitative test-set visualizations.","marker":"[10]"}],"fun_headline_variants":["Transformer flags bleeding in capsule endoscopy frames at 98% F1","End-to-end DETR model detects and classifies WCE bleeding","98% F1: one transformer net for GI bleeding detection and classification","Challenge third: transformer model achieves 98% F1 on WCE bleeding","Single transformer unifies bleeding detection and classification in WCE"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported metrics come from an 80:20 split of the challenge training set, and the paper assumes these numbers carry over to the independent test sets, for which it shows only qualitative sample images.","fun_headline_variants_meta":{"raw":{"variants":["Transformer flags bleeding in capsule endoscopy frames at 98% F1","End-to-end DETR model detects and classifies WCE bleeding","98% F1: one transformer net for GI bleeding detection and classification","Challenge third: transformer model achieves 98% F1 on WCE bleeding","Single transformer unifies bleeding detection and classification in WCE"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001061,"raw_usage":{"total_tokens":4466,"prompt_tokens":980,"completion_tokens":3486,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":3392}},"tokens_in":596,"tokens_out":3486,"duration_ms":24375,"temperature":1.0,"reasoning_tokens":3392,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:49:23.614455+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the released model on the official Auto-WCEBleedGen Test Sets 1 and 2 and compute classification accuracy and detection mAP; if these fall materially below 98.28% accuracy or 0.7328 mAP, the claimed transfer to unseen WCE data is refuted.","supporting_citations":[{"cited_title":"End-to-end object detection with transformers","cited_arxiv_id":null,"evidence_quote":"Defines the DETR architecture and the bipartite matching loss that the paper adapts for end-to-end detection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Hungarian assignment algorithm used in the bipartite matching loss."},{"cited_title":"Generalized intersection over union: A metric and a loss for bounding box regression","cited_arxiv_id":null,"evidence_quote":"Defines the (generalized) IoU loss component used for bounding box regression."},{"cited_title":"Detr: End-to-end object detection with transformers","cited_arxiv_id":null,"evidence_quote":"Provides the pretrained DETR/ResNet50 weights that make fine-tuning feasible on the small dataset."},{"cited_title":"Auto- wcebleedgen version v1 and v2: Challenge, datasets and evaluation","cited_arxiv_id":null,"evidence_quote":"Describes the Auto-WCEBleedGen challenge, its evaluation protocol, and the official competition context."},{"cited_title":"Autowcebleedgen-test dataset (improved), February 2024","cited_arxiv_id":null,"evidence_quote":"Is the independent test dataset used for the qualitative test-set visualizations."}],"review_version":1}