{"id":"59e2a2fa-2d91-4c8b-b643-b46e918e316c","arxiv_id":"2507.23088","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Perception Agent combines speech, large language models, and motion-based prompting to segment both known and novel surgical elements on demand.","lead":"This paper introduces Perception Agent, a system that lets surgeons control real-time segmentation of surgical tools and tissue by speaking and moving. A generalist may care because it demonstrates how combining speech recognition, large language models, and video segmentation could make surgical AI assistance more natural and adaptable.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The quantitative 'on par with manual prompting' claim is not actually about novel-element segmentation: Table 1 evaluates subsequent memory-based tracking of known instruments, never the initial object-centric mask, so the central claim for unseen elements lacks direct quantitative support.","rationale":"The reader's weakest assumption (motion dependence) is a real and acknowledged limitation of the method, but it is not the most load-bearing issue for the central claim. A more damaging concern is evidential: the only quantitative comparison in the paper measures memory-based tracking of known instruments, not the segmentation of unseen elements that the object-centric and reference-based mechanisms are designed to perform. The novelty of the work lies in segmenting novel elements via motion prompts, yet that step is never directly evaluated with a Dice or IoU; the figures in Table 1 are for downstream tracking where a stored memory may compensate for errors in the initial prompt-derived mask. The qualitative demonstrations are encouraging, but they do not provide the numerical support that the abstract claims. This does not change the reader's conditional verdict, but it sharpens the condition: either report direct metrics for the initial novel-element masks or explicitly limit the quantitative claim to memory-based tracking of known instruments.","tokens_in":7628,"tokens_out":8022,"duration_ms":93003,"concrete_test":"Compute the Dice and IoU of the initial object-centric segmentation masks (the masks stored as memory) against ground truth on the memory frame for the three instruments in Table 1, and similarly for reference-based novel segmentation on the dVRK videos if annotations can be added. If these direct metrics are not reported or are substantially lower than the tracking Dice reported in Table 1, the claim of on-par novel-element segmentation is unsupported by the current quantitative evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 compares the agent's segmentation performance using prior memory created either by manual prompting or by the object-centric approach. In both cases, the measured Dice (Table 1) is for memory-based tracking of instruments that already exist in the memory repository; the object-centric approach is used only to generate a single-frame mask that is then stored and injected into SAM2's memory bank. The paper never reports a Dice/IoU for that initial mask itself, i.e., the mask produced by prompting SAM2 with the motion-filtered points. Consequently, the central quantitative claim \"on par with manual prompting\" applies to tracking after a memory is available, not to the pipeline's capability to segment a previously unseen element. The novel-element mechanisms (object-centric and reference-based) are supported only by qualitative figures (Figs 4 and 5), which also rely on the motion assumptions acknowledged in the Discussion. This is a mismatch between the abstract's claim and the actual measurement.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Perception Agent, an interactive system that combines automatic speech recognition (Whisper), an LLM (GPT-4o), point tracking (CoTracker3), and SAM2 to segment surgical instruments and novel objects on demand. Two motion-based prompting mechanisms are proposed for unseen elements: object-centric segmentation (Eqs. 1-2) and reference-based segmentation (Eq. 3). The system stores masks in a memory repository for reuse in later surgeries. Experiments on EndoVis18 compare SAM2 tracking using prior memory created by manual prompting versus the object-centric approach, and qualitative dVRK experiments demonstrate segmentation of novel instruments, gauze, and phantom tissue grafts. The paper claims real-time, on-par performance with manual prompting.","tokens_in":7793,"tokens_out":6614,"duration_ms":76416,"significance":"If the central claims were fully supported, the work would be a useful systems-level demonstration of composing foundation models for interactive surgical perception, with a plausible path to reducing manual prompting burden. The use of a public benchmark, held-out test sequences, cross-dataset memory (EndoVis17 to EndoVis18), and qualitative dVRK demos are strengths. However, the quantitative evidence does not directly measure the novel-element segmentation step, the sample size is very small, real-time performance is not measured, and key parameters are unspecified. The contribution is therefore at the prototype/demonstration stage; the underlying idea is timely and worth further development, but the published claims need to be scaled back or supported by additional experiments.","major_comments":[{"comment":"The quantitative comparison does not measure the quality of the novel-element mask produced by the object-centric prompting mechanism. In both rows of the comparison study, the object-centric or manual prompt is used only to create a memory from a single frame, and the reported Dice/mIoU values are computed over subsequent frames while SAM2 tracks from that stored memory. The initial mask generated from the motion-filtered point set (Eqs. 1 and 2) is never evaluated against ground truth. Consequently, the abstract's claim that the agent performs 'on par with considerably more labor-intensive manual-prompting strategies' is only supported for memory-based tracking of already-known instruments, not for the segmentation of previously unseen elements. Please add an evaluation of the first-frame prompted mask, or a frame-wise evaluation starting from the initial prompt, and separately report the initial mask Dice/IoU. Also, the large gap between average Dice (0.96) and mIoU (0.62-0.74) suggests overprediction and should be analyzed rather than left unexplained.","section":"Section 3.1 / Table 1"},{"comment":"The parity claim rests on only three instruments from the EndoVis18 training set that happened to satisfy the motion criterion, with no error bars, confidence intervals, or significance testing. Since the object-centric approach is inherently limited to targets with significant coherent motion, the paper should quantify how often this condition holds in the dataset and report per-instrument variability. Without this, the numerical similarity between 0.963 and 0.965 (Table 1) is not a robust demonstration of parity.","section":"Section 3.1"},{"comment":"The real-time claim is unsupported. No latency, frame rate, or end-to-end runtime is reported anywhere; the pipeline includes speech recognition, an LLM call, dense point tracking, and SAM2 inference, each with substantial latency. Please include timing measurements for the speech-to-segmentation loop and for both novel-element mechanisms on the dVRK setup, or qualify the real-time claim as a design goal.","section":"Abstract / Sections 1 and 3"},{"comment":"The method's free parameters are not specified, which prevents replication and sensitivity analysis: the motion threshold gamma in Eq. (2) and Eq. (3), the number of query points Qp, the top-k matching points, the displacement threshold, and the tracking frame windows (16 and 48 frames) are all introduced without values. Please report the values used in the experiments and a sensitivity study for at least gamma and top-k.","section":"Section 2.2(c-d)"}],"minor_comments":[{"comment":"The title uses 'Interoperative', which appears to be a typo for 'intraoperative'; the text consistently uses 'intraoperative'.","section":"Title"},{"comment":"Table 1 is difficult to interpret because the first row reports all seven test instruments while the comparison rows report a three-instrument subset (T1, T3, T4, T5) with a different average; please restructure the table to distinguish the full test set from the subset and align column headings.","section":"Table 1"},{"comment":"Equations (2) and (3) define cosine similarity with a division by ||Vi(t)||, which is undefined for static points with zero motion; the manuscript should state how zero-motion tracks are handled in the computation.","section":"Section 2.2 / Eqs. (2)-(3)"},{"comment":"The phrase 'cherry-picked frame' is informal and the sentence contains a typo ('well-define'); please describe the frame-selection criterion objectively.","section":"Section 3.1"},{"comment":"The novel-element segmentation results are only qualitative; a small quantitative evaluation, such as manual annotation of a few frames, would strengthen the demonstration.","section":"Section 3.2 / Figs. 4-5"},{"comment":"No code, model weights, or demo video are provided; a reproducibility statement or public link would be helpful.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The main gap is the one identified in the quantitative evaluation: the headline parity claim is measured on downstream tracking, not on the initial novel-element segmentation. I believe this is fixable with additional experiments, so I recommend major revision rather than rejection. The manuscript would also benefit from a clearer statement that the system is an early prototype; the current abstract overstates real-time and on-par performance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take on arXiv:2507.23088. The interesting bit is the integration: speech + LLM parsing + SAM2 + CoTracker3, with a memory repository and two motion-prompting heuristics (object-centric and reference-based) to segment things not in the training vocabulary. That is genuinely new relative to the SAM-in-surgery literature, which typically uses manual clicks or text prompts. As an early prototype, the system makes sense and the qualitative demos (gauze, phantom grafts) show the idea can work under the right motion conditions.\n\nWhat it does well: the paper is honest about its own constraints. It explicitly says segmenting unseen static elements remains a challenge because there are no movement cues. It also reports that only three instruments in EndoVis18 met the motion criteria for the object-centric path, so it does not hide the selection bias. The citation pattern is appropriate; no red flags.\n\nThe soft spots are real, though. The stress-test holds up: the quantitative \"on par with manual prompting\" claim is about memory-based tracking of instruments that already have a memory. Table 1 measures Dice for downstream tracking when the prior memory was built by manual prompting versus by the object-centric method. It never reports the accuracy of the initial mask that the motion-prompted points produce. So the central claim for novel-element segmentation has no direct numbers. The mIoU values are also low (0.62–0.74), which suggests over-prediction even on the tracking task, and no error bars or per-frame variability are reported. The real-time claim is not backed by latency measurements. The free parameters (γ, k, Qp, thresholds, windows) are not stated or swept, so it's hard to know how sensitive the pipeline is.\n\nNone of these are fatal for a system paper, but they do mean the evidence is largely qualitative for the actual novelty. I don't see circularity – the memory comes from a training frame and is evaluated on held-out sequences.\n\nBottom line: a serious referee could get value out of this. It is a clear, readable description of a plausible architecture, and the gap between the abstract's claim and the measurement is worth pinning down in review. I would accept it for review with the expectation of major revisions on the evaluation. Not a paper I would build my own results on yet, but a fair contribution to the interactive-surgical-segmentation conversation.\n\nRegards","headline":"A clever agentic pipeline for hands-free surgical segmentation that is better motivated than measured; the headline quantitative claim tracks known tools, not novel-element segmentation.","tokens_in":8373,"tokens_out":2338,"would_cite":true,"duration_ms":24895,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Perception Agent lets surgeons segment known and never-seen objects in real time by voice, using motion-based prompting with a segment-anything tracker, and matches manual prompting accuracy on a public surgical benchmark.","keywords":["Perception Agent","surgical assistance","segmentation","Segment Anything Model","point tracking","large language models","human-machine interaction","motion-based prompting"],"falsifier":"Run the object-centric routine on a video where a never-before-seen instrument is held perfectly still while the scene is also still; the pipeline will have no trajectory signal to select prompt points and should fail to segment the object. To test the parity claim directly, compute the agent's Dice on all EndoVis18 test instruments using object-centric-derived memory and compare it to manual-prompt memory; a gap larger than a stated margin on clearly moving instruments would refute the claim of on-par performance.","tokens_in":7413,"feed_emoji":"🩺","tokens_out":5202,"duration_ms":63173,"temperature":0.7,"pith_summary":"This paper introduces Perception Agent, an AI system that lets a surgeon speak a request and have the agent segment the requested surgical element in the video stream. It claims the agent can handle both instruments it has seen before and objects it has never seen, such as custom grafts or gauze, by combining speech understanding with a large language model, a promptable segmentation model, and dense point tracking. On a public surgical video benchmark, the agent's segmentation accuracy was on par with manual point-prompting, averaging a Dice score of 0.963 versus 0.965. The proposed benefit is hands-free, on-demand assistance that adapts to whatever appears in the surgical field and remembers new objects for future surgeries.","feed_headline":"Voice-prompted AI agent matches manual surgical segmentation","feed_subtitle":"Agent tracks known tools and unseen grafts or gauze in real time, rivaling manual click prompting on benchmark video.","key_machinery":"The carrying mechanism is a coordinated pipeline of three foundation models anchored by a memory repository. SAM2, a promptable image-and-video segmentation model, generates masks from point prompts and keeps a memory bank of previous frames. CoTracker3, a dense any-point tracker, follows a grid of query points across video frames and produces the motion trajectories that drive novel-object discovery. The two novel prompting strategies compute per-point displacement and cosine similarity between trajectories, selecting points whose motion matches either the most strongly moving object in the scene or the reference object's motion template, then injecting those points as SAM2 prompts. In parallel, the agent stores a memory embedding of a newly segmented object in the repository so that later requests for the same object can be answered immediately without re-prompting.","core_discovery":"The central claim is that rigid, fixed-category surgical perception can be replaced by a flexible perception pipeline that treats the surgeon's spoken request as the prompt. Perception Agent transcribes speech, uses a prompt-engineered LLM to decide whether the requested element is already known or needs to be discovered, and then drives SAM2 with either a stored memory embedding or with point prompts derived from motion trajectories. For previously unseen elements, two mechanisms are proposed: object-centric segmentation, which selects tracked points that move most uniformly with the manipulated instrument, and reference-based segmentation, which selects candidate points whose motion matches a known reference object such as the needle driver holding a graft. The paper reports that on the EndoVis18 dataset, using memory created by the object-centric approach gives average Dice 0.963, comparable to 0.965 with manually prompted memory, and shows qualitative success on novel instruments, phantom tissue grafts, and gauze in simulated surgical settings.","pith_inferences":["The motion-dependence of the two novel mechanisms implies a testable boundary: static novel objects, or objects whose motion exactly matches the background, will not be discoverable; appearance-based or text-prompted discovery would be a natural next step.","The memory repository behaves like one-shot learning from interaction, and a hierarchical memory structure could encode part-whole relationships or surgical roles to make the agent more useful across related procedures.","Because the agent produces masks without manual annotation effort, the same prompting machinery could be reused to generate pseudo-labels for training downstream surgical models or to support instrument-pose estimation.","The reference-based mechanism, which compares candidate trajectories against a known reference object, could be extended to track deformable tissue whose motion is only partially coupled to the tool, though the paper does not quantify performance on such cases."],"forward_implications":["Surgeons can request tracking of a known instrument by name during an operation and receive real-time segmentation without manual clicking, because memory embeddings are reused across cases.","Objects never seen during training, including custom grafts, gauze, or tissue held by a tool, become segmentable on demand as long as they exhibit coherent motion relative to the scene or in sync with a reference object.","A novel element is memorized when first encountered, so in a later surgery the same element can be segmented directly from memory.","The approach matches manual-prompting accuracy on a public benchmark, suggesting that the added flexibility does not impose a clear segmentation-quality penalty for known instruments."],"supporting_citations":[{"why":"Supplies SAM2, the promptable segmentation model whose memory bank and mask output carry the actual object segmentation for known and novel elements.","marker":"[18]"},{"why":"Supplies CoTracker3, the dense any-point tracker whose trajectories provide the motion signal used to derive prompts for novel-object segmentation.","marker":"[14]"},{"why":"Provides the EndoVis18 public dataset used for quantitative comparisons of segmentation with prior memory from manual prompting versus the object-centric approach.","marker":"[1]"},{"why":"Provides the EndoVis17 dataset used to simulate memory from a previous surgery in the cross-surgery segmentation experiment.","marker":"[2]"},{"why":"Provides the da Vinci Research Kit used to create the simulated surgical videos for qualitative evaluation of novel-element segmentation.","marker":"[15]"},{"why":"Establishes the original Segment Anything model whose prompt-driven segmentation capabilities motivate and validate the use of SAM2 on surgical scenes.","marker":"[16]"}],"fun_headline_variants":["Voice-prompted AI agent rivals manual surgical segmentation","Speech-driven AI learns to segment novel tools and gauze in surgery","Flexible surgical AI adapts to unseen objects via voice prompts","Real-time speech AI matches manual prompting for surgical segmentation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The agent only finds a novel object by motion: if the target object does not move coherently on its own or in sync with a reference tool, the motion filters cannot select points to prompt the segmenter, so unseen static objects cannot be segmented.","fun_headline_variants_meta":{"raw":{"variants":["Voice-prompted AI agent rivals manual surgical segmentation","Speech-driven AI learns to segment novel tools and gauze in surgery","Flexible surgical AI adapts to unseen objects via voice prompts","Real-time speech AI matches manual prompting for surgical segmentation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000272,"raw_usage":{"total_tokens":1652,"prompt_tokens":987,"completion_tokens":665,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":597}},"tokens_in":603,"tokens_out":665,"duration_ms":7778,"temperature":1.0,"reasoning_tokens":597,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:03:28.083431+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the object-centric routine on a video where a never-before-seen instrument is held perfectly still while the scene is also still; the pipeline will have no trajectory signal to select prompt points and should fail to segment the object. To test the parity claim directly, compute the agent's Dice on all EndoVis18 test instruments using object-centric-derived memory and compare it to manual-prompt memory; a gap larger than a stated margin on clearly moving instruments would refute the claim of on-par performance.","supporting_citations":[{"cited_title":"In: 2014 IEEE international conference on robotics and automation (ICRA)","cited_arxiv_id":null,"evidence_quote":"Provides the da Vinci Research Kit used to create the simulated surgical videos for qualitative evaluation of novel-element segmentation."}],"review_version":1}