{"id":"5c499ff1-8832-4553-a521-c8e37c0e16d0","arxiv_id":"2501.15994","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A YOLO11-based detector identifies brain tumors in intraoperative ultrasound in real time and was qualitatively validated in 15 consecutive surgeries.","lead":"Researchers trained YOLO11 object-detection models to spot brain tumors in intraoperative ultrasound images, reaching a mean average precision of 0.95 and running at 34 frames per second. They then deployed the model live in 15 brain tumor surgeries, where surgeons judged the real-time tumor outlines to be correct and unobtrusive.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set labels are not independent: 1,535/1,732 images use expert-refined nnU-Net pseudo-labels, and the same nnU-Net saw one manual slice per test patient; reported mAP may overstate performance against true expert annotations.","rationale":"Good-faith reading: the paper is an engineering and clinical feasibility study, not a claim of mathematical novelty. Its central quantitative evidence is the YOLO11 detection benchmark in Tables 3 through 5, plus a qualitative 15-patient OR deployment. For the quantitative claim to hold, the test labels must be a valid proxy for expert truth. Section 3.2 shows this condition is not met cleanly: about 89% of images (1,535/1,732) have labels generated by the authors' own nnU-Net, then expert-refined, and the nnU-Net itself was trained on one manual slice per patient, including the test patients. The test set is subject-stratified but still mostly pseudo-labeled. This is not a fatal flaw, because expert refinement and the previously reported nnU-Net Dice (0.9 internal, 0.93 external) provide some support; however, the reported mAP numbers are conditional on the pseudo-labels being unbiased. The prospective OR evaluation is qualitative and self-assessed, so it does not independently verify mAP. The single most load-bearing check is therefore a fully manual, independent re-annotation of the test set and recomputation of the headline metrics. The paper's other contributions - multicenter data, real-time FPS, and systematic variant comparison - stand independent of this concern. The reader's verdict of CONDITIONAL is appropriate; no change is needed.","tokens_in":20271,"tokens_out":4439,"duration_ms":40266,"concrete_test":"Take the 359 test-set images (or a stratified random subset of at least 100, preserving the center, tumor-type, and size mix) and have two neurosurgeons who did not generate the pseudo-labels, and who are blinded to both the nnU-Net outputs and the YOLO11 predictions, independently draw tumor bounding boxes and segmentations using the same protocol as Section 3.2; resolve disagreements by consensus. Then recompute Table 3's mAP@50, mAP@50-95, precision, and recall for YOLO11s against these fully manual labels. If mAP@50 falls below about 0.90 or mAP@50-95 falls below about 0.55 (a drop larger than the reported confidence intervals), the pseudo-label ground truth is inflating the headline numbers and the performance claims should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claims (Section 6.1, Table 3) rest on test labels that are not independent of the label-generation process. Section 3.2 states that 1,535 of the 1,732 images were labeled by the authors' previously published nnU-Net model, then reviewed and refined by two neurosurgeons; only 197 images have fully manual segmentations. Section 5.6 splits the 1,732 images 70/10/20 with subject stratification, so the 359-image test set consists mostly of pseudo-labeled images. Because YOLO11 was trained on the same pseudo-labeled targets it is later scored against, any systematic error in the nnU-Net labels (e.g., consistent over- or under-inclusion of peritumoral edema, or bias toward one representative slice orientation) can be learned by YOLO11 and then counted as correct in evaluation. The expert refinement reduces but does not eliminate this risk, and no inter-observer agreement or correction statistics are reported. There is an additional subtlety: the nnU-Net was trained on one manual slice per patient; since the test subjects are included in that set, the pseudo-labels for test images inherit patient-specific information, making them even less like fresh expert annotations. The prospective OR results (Section 6.3) are qualitative and were scored by the same surgeons who built the system, so they do not independently calibrate the mAP figures. The load-bearing assumption is therefore that expert-refined pseudo-labels are unbiased enough to serve as ground truth; Section 7.1 acknowledges manual-segmentation variability but does not address this specific circularity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a real-time brain tumor detection system for intraoperative ultrasound (ioUS) using YOLO11, trained on 1,732 images from the BraTioUS and ReMIND datasets. Most labels are pseudo-labels produced by the authors' earlier nnU-Net segmentation model and subsequently refined by two neurosurgeons; data augmentation expands the training set to 11,570 images. The authors evaluate five YOLO11 variants for both object detection and instance segmentation, reporting for YOLO11s a mAP@50 of 0.95, mAP@50-95 of 0.65, and 34.16 FPS. They also report a prospective deployment in 15 patients, with surgeons qualitatively confirming that real-time predictions delineated tumor regions, including a few residual-tumor detections during resection. The paper claims this is the first real-time object detection implementation for ioUS-guided brain tumor surgery.","tokens_in":20537,"tokens_out":5180,"duration_ms":50019,"significance":"If the quantitative results are correct, the paper would make a useful engineering contribution: it provides a multicenter benchmark for YOLO11 variants on ioUS, demonstrates a low-cost operating-room deployment pipeline, and identifies a clinically relevant use case in residual tumor detection. The authors are explicit that segmentation is secondary, and their bounding-box detection framing is appropriate for real-time surgical workflow. The dataset size and multicenter coverage are strengths. However, the central quantitative claims in Table 3 and Section 6.1 rest on test labels that are not independent of the training-label generation process: 1,535 of 1,732 images have labels generated by the authors' own prior model. The expert refinement reduces but does not quantify the resulting bias. The prospective validation in Section 6.3 is qualitative, unblinded, and self-assessed by the system developers, so it does not independently calibrate the reported mAP values. The paper's own limitation section discusses interobserver variability but does not address the pseudo-label circularity, which is the more serious threat to the reported accuracy.","major_comments":[{"comment":"The test-set ground truth is not independent of the label-generation process. The paper states that only 197 of 1,732 images have manual segmentations, while the remaining 1,535 images are pseudo-labels generated by the authors' previously published nnU-Net model and then refined by two neurosurgeons. Because these same pseudo-labels define both the training target and the benchmark for the 359-image test set, any systematic bias in the nnU-Net labels (e.g., consistent inclusion or exclusion of peritumoral edema, or a bias toward particular slice orientations) can be learned by YOLO11 and then counted as correct during evaluation. Moreover, since the nnU-Net was trained on one manual slice per patient and the test patients are drawn from the same cohort, the pseudo-labels for test images inherit patient-specific information that makes them even less like fresh expert annotations. The reported mAP@50 of 0.95 and mAP@50-95 of 0.65 therefore depend critically on an unverified assumption that the expert-refined pseudo-labels are unbiased enough to serve as ground truth. The manuscript should provide a held-out manually annotated test set, or at minimum report inter-observer agreement, correction statistics, and performance stratified by manual versus pseudo-labeled test images.","section":"Section 3.2 and Section 5.6 with Table 3"},{"comment":"The prospective 'validation' is entirely qualitative and was performed by the same surgeons who built the system. The abstract claims that 'real-time predictions accurately delineated tumor regions,' but the study reports no quantitative metric, no pre-specified evaluation criteria, no blinding, and no independent reference such as postoperative MRI or histopathology. The residual-tumor anecdotes for Patients 2, 11, and 13 are suggestive but uncontrolled and are evaluated by the developers themselves. As written, the prospective results do not independently calibrate the mAP figures and do not establish clinical benefit. The claims of clinical validity should be softened, or the trial design should be extended with protocol-based independent assessment.","section":"Section 6.3 and Table 2"}],"minor_comments":[{"comment":"The text says the warmup strategy was implemented 'during the first 3 seasons'; this should read 'epochs.'","section":"Section 5.5"},{"comment":"The column header 'DCS' is used, but the definition and text use 'DSC' (Dice similarity coefficient). The notation should be consistent.","section":"Table 6"},{"comment":"The recall confidence interval for YOLO11l is reported as (0.95, 1.01), which exceeds the maximum possible value of 1.0 for a proportion. This suggests an issue in the confidence interval computation, or the CI description should specify the resampling method and any clipping convention.","section":"Table 3"},{"comment":"The text and table totals are consistent (197 subjects), but the MGH column lists age and sex as NA; please clarify whether these variables are unavailable for all five MGH patients and state this explicitly in the table notes.","section":"Section 3.1 and Table 1"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the pseudo-label circularity is the central threat to the paper's quantitative claims. The authors' own prior nnU-Net generates most training and test labels, and the one manual slice per patient used to train that model comes from the same cohort as the test subjects. This is a load-bearing issue that cannot be resolved by rephrasing alone; the authors need an independent manually annotated test set or a clearly reported and guarded comparison between manual and pseudo-labeled subsets. Given the engineering contribution and the realistic deployment effort, I prefer major revision over rejection, but if the authors cannot provide such evidence, the manuscript's reported accuracy numbers should not be published as they now stand."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, the short version: this is a legit engineering study with a real evaluation flaw. The authors deploy a YOLO11 detector for brain tumors in intraoperative ultrasound, run it live in 15 surgeries, and show it runs at 34 FPS on a laptop. That part is new and useful. The headline mAP numbers (0.95 @50, 0.65 @50-95) should be read as optimistic because 1,535 of 1,732 images carry pseudo-labels from the authors' own nnU-Net, and those same pseudo-labels are both the training target and the test benchmark. The expert refinement helps, but there's no inter-observer agreement, no correction statistics, and the nnU-Net was trained on the test patients' own manual slices, so the test set is partly self-agreement. Credit where due: the multicenter dataset (six centers, 197 patients) is a real asset, the YOLO variant comparison is thorough, and the prospective OR work is the first of its kind for this modality. The residual tumor case in patient 11—model flagged something the surgeon hadn't seen, and re-inspection confirmed it—is the kind of anecdote that motivates clinical trials, even though it's n=1. The weaknesses are proportional. The OR evaluation is qualitative and self-assessed by the same team that built the system, so it tells you the workflow doesn't break, not whether the detections are more accurate than the surgeon. The paper's own limitation section mentions manual segmentation variability but doesn't confront the pseudo-label circularity. That's a gap in the argument, and the reviewer should force it open. Bottom line: the real-time feasibility claim holds; the accuracy claim does not, as stated. This deserves a serious referee—it's a genuine first step with a usable dataset and a clear path to external validation—but acceptance should be conditional on fully manual test labels or an independent test cohort, and on tempering the language about accuracy. If you work in surgical ultrasound, cite it for the deployment experience and the dataset, not for the mAP.","headline":"A genuine first deployment of real-time brain tumor detection in intraoperative ultrasound, worth refereeing but the accuracy claims are weakened by pseudo-label circularity in the test set.","tokens_in":21185,"tokens_out":2165,"would_cite":true,"duration_ms":22088,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Real-time YOLO11 object detection can locate brain tumors in intraoperative ultrasound and flag residual tumor during resection, with the best model reaching mAP@50 of 0.95 at 34.16 frames per second.","keywords":["intraoperative ultrasound","brain tumor","YOLO11","object detection","real-time detection","glioma surgery","image-guided neurosurgery","pseudo-labels"],"falsifier":"Re-annotate the 359-image test set with independent manual outlines from neurosurgeons who did not produce the pseudo-labels, then recompute mAP@50; if the score drops materially below the reported 0.95, the shared label-generation process inflated the result.","tokens_in":20044,"feed_emoji":"🧠","tokens_out":9410,"duration_ms":79569,"temperature":0.7,"pith_summary":"This paper tries to establish that a real-time object-detection model can make intraoperative ultrasound interpretable enough to guide brain tumor surgery. The authors trained YOLO11 variants on 1,732 ultrasound images from 192 patients, augmented to 11,570 images, and report that the small variant detects tumors with mAP@50 of 0.95 at 34.16 frames per second and 24.9 ms latency. They then deployed the model live in 15 consecutive craniotomies, where surgeons judged the bounding boxes correct and, in one case, the model flagged a residual tumor the surgeon had not initially seen. If true, this would give low-cost ultrasound a computer-vision assist that currently requires expensive intraoperative MRI, and could help surgeons maximize safe resection of gliomas.","feed_headline":"YOLO11 spots brain tumors on ultrasound at 34 frames per second","feed_subtitle":"The model flagged residual tumor in 15 live surgeries, including one the surgeon had not noticed.","key_machinery":"The central object is YOLO11, a single-pass convolutional object detector that divides the image into a grid and predicts bounding boxes and class probabilities directly from the whole image. Its backbone, neck, and head use the C3k2 block, fast spatial pyramid pooling, and a parallel spatial attention module that together provide multi-scale feature extraction cheaply enough for real-time inference. The carrying mechanism for the training data is a pseudo-label pipeline: a prior segmentation model generated tumor outlines for 1,535 of the 1,732 images, and two expert neurosurgeons refined them, so the detector's training target and its test benchmark come from the same label-generation process. Pretrained weights from a large general object dataset and data augmentation, including rotations, flips, shears, brightness shifts, blur, and noise, expand the training set to 11,570 images.","core_discovery":"On its own terms, the paper's central discovery is that a single-pass detector, the small YOLO11 variant, balances precision and speed well enough for the operating room: mAP@50 of 0.95, mAP@50-95 of 0.65, recall and precision above 0.93, a 24.9 ms latency, and 34.16 frames per second on a laptop GPU. The same training pipeline also produced instance segmentation with a Dice score of 0.88, but the paper treats detection as the primary task because surgeons need to know where the tumor is rather than its exact pixel contour. The prospective deployment showed the model tracking tumors across scanners and tumor types, including meningiomas that were absent from training, and in three patients it detected residual tumor during resection, including one remnant confirmed only after the surgeon re-inspected the cavity.","pith_inferences":["Editorial inference: because the five YOLO variants score nearly identically on mAP@50, between 0.93 and 0.95, the detector's ceiling is likely set by label fidelity and case diversity rather than architecture size; a testable prediction is that improving the pseudo-labels will raise mAP more than switching to a larger variant.","Editorial inference: adding post-resection images and Doppler-mode images to the training set is a direct next step, since the prospective cases already show the model working on both input types; training on them should sharpen residual-tumor detection specifically.","Editorial inference: the same detection-first design could transfer to other ultrasound-guided procedures where real-time localization is the bottleneck, provided a similarly curated multicenter label pipeline is available.","Editorial inference: a clean way to quantify label-induced bias would be to train and evaluate the same YOLO11 pipeline on a fully manually annotated subset of equal size and compare mAP against the pseudo-label-trained model."],"forward_implications":["A laptop with a mid-range GPU can run the detector at 34.16 frames per second, so the system adds no operative delay and requires no expensive dedicated imaging hardware.","The detector can flag residual tumor during resection; in one prospective case it found a remnant that the surgeon initially did not see, suggesting it could increase the extent of resection.","Bounding-box detection, rather than pixel-level segmentation, is sufficient for intraoperative guidance, which lowers the engineering cost of deploying such tools.","The model ignores scanner interface graphics such as menus and buttons in raw video, so it can be connected directly to the ultrasound output without preprocessing.","The detector generalized to tumor types absent from training, such as meningiomas, though for some meningioma cases it included perilesional edema inside the bounding box."],"supporting_citations":[{"why":"Defines the single-pass YOLO detection paradigm that the paper adapts to intraoperative ultrasound.","marker":"[17]"},{"why":"Supplies the YOLO11 implementation, its five variants, and the pretrained weights used for fine-tuning.","marker":"[42]"},{"why":"Describes the YOLO11 architectural components that the paper credits for its efficiency and accuracy.","marker":"[40]"},{"why":"The authors' earlier segmentation model whose outputs became the pseudo-labels for 1,535 of the 1,732 images.","marker":"[18]"},{"why":"The self-configuring segmentation method used to build that earlier model and generate the pseudo-labels.","marker":"[33]"},{"why":"Public resection dataset that contributed 45 of the 197 subjects and a subset of the images.","marker":"[32]"},{"why":"Large general object-detection dataset whose pretrained weights initialize all YOLO11 variants before fine-tuning on ultrasound images.","marker":"[44]"},{"why":"Platform used for preprocessing and data augmentation, expanding the training set to 11,570 images.","marker":"[45]"}],"fun_headline_variants":["YOLO11 detects brain tumors in real time at 34 fps","Live brain tumor spotting: YOLO11 hits 34 fps in surgery","YOLO11 catches tumor remnants surgeons missed during live surgery","Small YOLO11 model tracks brain tumors at 34 fps in OR","Real-time brain tumor detection: YOLO11 achieves 34 fps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracy assumes that the machine-generated tumor outlines, refined by experts for 1,535 of the 1,732 images, are accurate enough to serve as ground truth for both training and testing; if those outlines carry systematic bias, the mAP numbers overstate real detection performance.","fun_headline_variants_meta":{"raw":{"variants":["YOLO11 detects brain tumors in real time at 34 fps","Live brain tumor spotting: YOLO11 hits 34 fps in surgery","YOLO11 catches tumor remnants surgeons missed during live surgery","Small YOLO11 model tracks brain tumors at 34 fps in OR","Real-time brain tumor detection: YOLO11 achieves 34 fps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000569,"raw_usage":{"total_tokens":2734,"prompt_tokens":1027,"completion_tokens":1707,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":643,"completion_tokens_details":{"reasoning_tokens":1611}},"tokens_in":643,"tokens_out":1707,"duration_ms":11409,"temperature":1.0,"reasoning_tokens":1611,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:49:23.898652+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate the 359-image test set with independent manual outlines from neurosurgeons who did not produce the pseudo-labels, then recompute mAP@50; if the score drops materially below the reported 0.95, the shared label-generation process inflated the result.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The authors' earlier segmentation model whose outputs became the pseudo-labels for 1,535 of the 1,732 images."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The self-configuring segmentation method used to build that earlier model and generate the pseudo-labels."},{"cited_title":"Ultralytics YOLO","cited_arxiv_id":null,"evidence_quote":"Platform used for preprocessing and data augmentation, expanding the training set to 11,570 images."}],"review_version":1}