{"id":"ce3b1dc4-7955-41d0-8db3-e4b27c26c6db","arxiv_id":"2411.18977","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A fully automated pipeline that pairs YOLOv8 detections with SAM2 to segment infinite videos at constant memory, with additional engineering for online object addition and preloaded memory banks.","lead":"Det-SAM2 is a self-prompting video segmentation pipeline that combines a YOLOv8 detector with SAM2, automatically generating segmentation prompts so infinite-length video streams can be processed without manual input. This matters because it brings SAM2's segmentation power to fully automated, memory-constrained industrial deployments like AI refereeing in billiards.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Bounded-window propagation makes 'same accuracy as SAM2' unsubstantiated: corrections to released frames are impossible, and §5.2 admits long-association-span degradation without any window-size evaluation.","rationale":"The reader's weakest_assumption identifies the bounded-window issue, and I agree with it; in fact, it can be sharpened. It is not merely missing verification: §3.7's release_old_frames makes late corrections to old frames impossible by design, and §5.2 concedes the limitation. The abstract's equality-of-accuracy statement is therefore internally at odds with the described system unless the retention window is always large enough for the video's association span. Since §4 provides only qualitative visualizations and no benchmark, the claim cannot be evaluated as stated. The engineering contributions (online ID addition, preloaded memory, offloading, frame release) appear real and are described at the code level; I am not challenging their existence. I therefore do not ask for rejection, only for the conditional requirements already proposed: quantitative memory and accuracy measurements with a window-size sweep, and a commit-pinned code release. Since the reader already set CONDITIONAL, my verdict is unchanged.","tokens_in":10452,"tokens_out":4409,"duration_ms":43664,"concrete_test":"Construct a synthetic video (or adapt the billiards setup) where a tracked object exits the frame for T frames and reappears, and where the detector fails to prompt in the first R frames after reappearance. Run Det-SAM2 with max_inference_state_frames = 4, 8, 16, 64 and full memory, and compare per-frame mask IoU against original SAM2 run with a correction prompt at the reappearance frame. If masking accuracy on frames older than the retention window degrades as the window shrinks, or if the late correction fails to restore masks that were already released, the constant-memory/equal-accuracy claim fails. Log peak VRAM/RAM over a 10,000-frame stream to confirm whether memory is truly flat.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the abstract's assertion that Det-SAM2 enables inference on infinitely long video streams with constant VRAM/RAM usage 'while preserving the same efficiency and accuracy as the original SAM2.' The load-bearing condition is that corrective information never needs to reach a frame older than the retention window. SAM2's correction mechanism works by propagate_in_video over all historical frames (§3.2); Det-SAM2 truncates this to max_frame_num_to_track (§3.4) and then physically releases older frames via release_old_frames at max_inference_state_frames (§3.7). After release, no late prompt can alter a released frame. So Det-SAM2 is not a memory-neutral implementation of SAM2 on unbounded streams; it is SAM2 with a finite-horizon correction model. The paper acknowledges exactly this in §5.2, noting that constraining the memory bank 'will inevitably have an impact on videos with a large object association span,' yet it leaves open whether any finite window preserves 'the same accuracy.' Since §4 contains no quantitative comparison against SAM2 and no memory/accuracy curve as a function of window size, the abstract's guarantee is unsupported for the regime where corrections span beyond the window. This is the weakest point because every other promised property (constant memory, online IDs, preloaded memory) is either implementation-level or contingent on this locality assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Det-SAM2, a fully automated video segmentation pipeline that replaces SAM2's manual point/box prompts with prompts generated by a YOLOv8 detector. The authors describe several engineering modifications: accumulating video frames into buffers, limiting the propagation range of SAM2's correction mechanism, preloading a memory bank from previous videos, supporting online addition of new object IDs, and continuously releasing old frames and cached data to bound VRAM/RAM usage. The stated contribution is that the pipeline enables inference on infinitely long video streams with constant memory usage while preserving SAM2's efficiency and accuracy. The report includes code snippets, schematic figures, and a billiards application example, but contains no quantitative experiments.","tokens_in":10793,"tokens_out":3658,"duration_ms":35042,"significance":"If the claims are correct, the pipeline is practically valuable: it would allow automated, memory-bounded video segmentation on unbounded streams, with accuracy inherited from SAM2 and the ability to adapt to new objects at runtime. The engineering changes are described in enough detail that they could be reproduced from the referenced code repository. However, the central empirical claims—constant memory usage and accuracy equal to SAM2—are not demonstrated by any quantitative evaluation, and the acknowledged limitation of a finite correction window directly challenges the accuracy claim.","major_comments":[{"comment":"The experimental section contains no quantitative evaluation at all. The abstract claims constant VRAM/RAM usage and accuracy equal to SAM2, but the paper reports no memory measurements, no runtime measurements, and no segmentation accuracy metrics (e.g., mIoU, J&F, or a comparison with SAM2). The only evidence is a visualization of a billiards scene. This is load-bearing for the central claims. Please add memory profiling over long videos, wall-clock time, and accuracy comparisons against SAM2 (and ideally against a baseline without the memory-release and propagation-limiting modifications), with explicit settings for max_frame_num_to_track, max_inference_state_frames, and detection interval.","section":"Section 4, Experiments"},{"comment":"The paper acknowledges that constraining the memory bank to recent frames 'will inevitably have an impact on videos with a large object association span.' This means the claim of 'preserving the same accuracy as the original SAM2' is not generally true: SAM2's propagation mechanism can propagate corrections across all historical frames, whereas Det-SAM2 truncates this to max_frame_num_to_track and then physically releases older frames via release_old_frames. No analysis or experiment quantifies how accuracy degrades as the association span grows, nor does the paper provide guidance on choosing window sizes. To support the accuracy claim, please include an accuracy-versus-window-size study on videos with varying association spans, or revise the claim to reflect the bounded-correction nature of the system.","section":"Section 5.2"},{"comment":"The constant-memory claim depends on immediately releasing entries from video_segments after post-processing, as shown in Section 3.7(7). However, Appendix D states that the post-processing thread 'may reprocess previously processed frames.' If frames are popped from video_segments, reprocessing requires re-running inference or retaining the data, which undermines the constant-memory guarantee. Please clarify how reprocessing and frame release coexist, and provide actual memory-usage curves demonstrating constant RAM/VRAM over an extended video stream.","section":"Section 3.7 and Appendix D"}],"minor_comments":[{"comment":"There is a typographical error: 'YOLOv8[3] detection model,This pipeline' should have a period and space after the comma. Also, the phrase 'same efficiency and accuracy' is ambiguous; the paper later notes a ~22% inference-time increase when offload_state_to_cpu=True, so 'same efficiency' needs a precise definition (e.g., per-frame throughput or total time).","section":"Abstract and Section 1"},{"comment":"Equation and variable naming: 'By limiting the propagation length( max_frame_num_to_track) to M, and the cumulative frame buffer size (max_frame_num_to_track) to K' repeats the same parameter name for two different quantities. Please introduce a distinct name for the buffer size (e.g., buffer_size or K) throughout the paper.","section":"Section 3.3"},{"comment":"The formula for computational cost '1/2 N^2' and 'M/K N' would be clearer with parentheses (e.g., (1/2)N^2 or N^2/2, and (M/K)N). Also, references [5] and [6] currently share the same URL; if they are different comments, please provide distinct URLs and comment IDs.","section":"Section 3.7 and References"},{"comment":"Figure 12's caption says 'is shown in Figure 13,' but Figure 13 appears to be a different rendering of the same result. Please clarify what each figure shows and ensure the captions are self-contained.","section":"Figure 12/13 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper is more of an engineering technical report than a full research paper, which is fine for some venues, but the absence of any quantitative evaluation is a serious gap for a journal submission. The code availability is a strength, and with a modest set of experiments (memory profiling, accuracy ablation over window sizes, runtime comparison) the central claims could be substantiated. If the authors cannot provide such experiments, the claims should be scaled back to match what is demonstrated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The genuinely useful part is the engineering: Det-SAM2 wires a detector to SAM2, adds online object-ID registration so you don't have to reset the memory bank, lets you preload memory from a previous video, and caps propagation while releasing old frames to keep memory bounded. Those are concrete changes, and the paper is candid about where official SAM2 falls short. The code is referenced, and the appendices give enough detail to reproduce the main modifications. That's real value.\n\nThe soft spot is exactly where the stress-test note points. The abstract says \"preserving the same efficiency and accuracy as the original SAM2\" on infinitely long streams. That is not supported. Section 4 is visual only. No memory curves, no VRAM/RAM numbers, no accuracy comparison against SAM2, no window-size sweep. And the paper's own Section 5.2 admits that bounding the memory bank hurts long association spans. Once release_old_frames() drops a frame, a later correction cannot reach it, so Det-SAM2 is SAM2 with a finite-horizon correction model, not a memory-neutral implementation. The claim may be true for many real streams, but it hasn't been shown.\n\nI also note that the memory claims read as if they were measured, but the only concrete numbers are per-frame estimates from the official offload flags. The constant-memory conclusion follows from the release logic, which is plausible, but \"constant\" plus \"same accuracy\" needs a benchmark. Minor: Section 3.4 has a notation slip, labeling both the propagation limit and the buffer size as max_frame_num_to_track; presumably one of them is the frame buffer size.\n\nOn citations: the paper leans on SAM2, YOLOv8, and GitHub issues. That's appropriate for an engineering report, and the issue attributions are actually helpful. No invented entities, no parameter fitting to data, so the circularity concern doesn't land.\n\nVerdict: useful technical report, not a paper with demonstrated results. It deserves engagement as a system description, and the code could save someone real time. But the central claims need measurements, a commit-pinned repo, and a test of the bounded-window assumption. I would not send this to a formal peer-reviewed venue in its current state; for a workshop or arXiv-style technical report, fine.","headline":"Honest engineering write-up with genuinely useful memory-management ideas, but the abstract's guarantee of SAM2-equal accuracy on infinite streams is unsupported by any measurement.","tokens_in":11232,"tokens_out":2642,"would_cite":false,"duration_ms":26580,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a detection model can take over SAM2's prompting, turning it into an automatic pipeline that segments unlimited video with constant memory and unchanged accuracy.","keywords":["video instance segmentation","Segment Anything Model 2","SAM2","self-prompting pipeline","constant memory inference","object detection prompts","long-video tracking","memory bank optimization"],"falsifier":"Run a long video through Det-SAM2 with a small max_inference_state_frames, verify an early segmentation mistake, then feed a corrective prompt after that frame has been released and compare the early mask with the result from a run using unbounded propagation; if the early mask stays wrong, the bounded-memory pipeline has not preserved SAM2's correction behavior, contradicting the central accuracy claim.","tokens_in":10205,"feed_emoji":"🎱","tokens_out":8366,"duration_ms":70433,"temperature":0.7,"pith_summary":"SAM2 is a video segmentation model whose official workflow needs a human-provided prompt on the first frame and restarts its inference whenever a new prompt or object category arrives. This report claims that a detection model can take over prompt generation so that no human is needed, and that several memory-bank controls can make the cost per frame independent of video length. The resulting pipeline, Det-SAM2, is said to segment infinitely long video streams with constant VRAM and RAM usage while keeping SAM2's accuracy, to accept new object IDs during tracking, and to carry memory from one video into the next. The practical payoff is that an interactive research tool becomes an unattended real-time system, demonstrated here as an AI referee for billiards, without retraining SAM2.","feed_headline":"Detector prompts let SAM2 run forever with fixed memory","feed_subtitle":"A self-prompting wrapper streams unbounded video through SAM2 without growing RAM or GPU use, keeping mask quality.","key_machinery":"The load-bearing object is the SAM2 inference state, which bundles per-frame image features, mask logits, condition-frame outputs, object-ID mappings, and the memory bank used by memory attention. Det-SAM2 works by capping and recycling that state: propagation runs backward over at most max_frame_num_to_track frames, release_old_frames drops frames older than max_inference_state_frames while protecting preloaded-memory frames, images are offloaded to CPU and later deleted through an images_idx index, and adding a new object ID re-encodes recent memory under an expanded ID mapping. The preloaded memory bank acts as a fixed starting memory that is never released. These bounds make the working set depend on the window sizes rather than on total video length, trading correction range for constant resource use.","core_discovery":"The paper's central claim is that the interactive SAM2 video predictor can be wrapped in an automatic pipeline purely by engineering. A detection model emits bounding-box prompts, which are fed to the SAM2 prompt encoder on selected frames; SAM2 then propagates masks and can retroactively correct earlier frames, but only within a bounded recent-past window controlled by the pipeline. Old frames and cached features are released once they fall outside that window, an index mapping keeps the remaining tensors consistent, and the memory bank is updated online when new object IDs appear. A preloaded memory bank built from an earlier video can serve as a \"system prompt\" for a new video, removing the need for any prompt on that new stream. The stated consequence is that a stream of length $N$ costs roughly $\\frac{M}{K}N$ frame-processings rather than $\\frac{1}{2}N^2$, with constant memory, while the billiards application shows the masks feeding goal, collision, and cushion-rebound detection.","pith_inferences":["The paper leaves the retention-window size unquantified; its own discussion concedes that long object-association spans may suffer, so a natural test is whether accuracy drops when an object disappears and reappears after more than max_inference_state_frames.","One extension is to replace the bounded memory bank with a fixed-parameter state representation of the kind the paper names as future work, which could keep full-history context under constant memory.","The detector-to-SAM2 ID gap means the current pipeline handles at most one instance per category per frame; generalizing to multiple instances of the same class would require an external matching or tracking step, a limitation the paper explicitly leaves open.","The constant-memory mechanism can be validated independently by instrumenting RAM and VRAM on a synthetic stream while toggling release_old_frames; any growth with stream length would reveal a residual leak."],"forward_implications":["If the claim holds, live camera streams can be segmented indefinitely with a memory footprint fixed by the retention window instead of the stream length.","Segmentation quality for categories the detector can see stays at SAM2's level, including handling of deformation and occlusion, because the underlying model is unchanged.","New object categories can appear midstream without resetting the memory bank, so unexpected objects do not erase earlier tracking results.","A memory bank carried over from one video can warm-start the next video, letting a new stream start with no prompts at all.","Speed and correction range can be tuned by choosing the detection interval and the maximum propagation window, giving operators a practical accuracy-latency dial."],"supporting_citations":[{"why":"SAM2 is the base model whose video predictor, memory bank, memory attention, and propagate_in_video correction mechanism Det-SAM2 builds on and modifies.","marker":"[1]"},{"why":"The YOLOv8 detection model supplies the automatic bounding-box prompts that replace human condition prompts.","marker":"[3]"},{"why":"A thread in the SAM2 codebase suggesting that a memory bank built for one video can be reused for another, giving rise to the preload memory bank feature.","marker":"[5]"},{"why":"A thread on reducing GPU memory overhead that inspired the release_old_frames and cache-clearing strategy.","marker":"[6]"},{"why":"A thread suggesting FP16 storage for video frames, used to reduce per-frame memory by roughly 0.007 GB.","marker":"[7]"},{"why":"The RWKV-style fixed-parameter state paper cited as the future direction for replacing the bounded memory bank with full-context constant-memory state.","marker":"[8]"}],"fun_headline_variants":["Det-SAM2: auto-prompts for endless video segmentation","Fixed-memory video segmentation with self-prompting SAM2","Detector prompts let SAM2 run infinite video streams","SAM2 without prompts: constant memory for long videos"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that corrective information never has to reach frames older than the configured retention window; if a late prompt would fix a mistake in an already-released frame, that error becomes permanent.","fun_headline_variants_meta":{"raw":{"variants":["Det-SAM2: auto-prompts for endless video segmentation","Fixed-memory video segmentation with self-prompting SAM2","Detector prompts let SAM2 run infinite video streams","SAM2 without prompts: constant memory for long videos"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000198,"raw_usage":{"total_tokens":1357,"prompt_tokens":923,"completion_tokens":434,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":367}},"tokens_in":539,"tokens_out":434,"duration_ms":4799,"temperature":1.0,"reasoning_tokens":367,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:39:07.259637+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a long video through Det-SAM2 with a small max_inference_state_frames, verify an early segmentation mistake, then feed a corrective prompt after that frame has been released and compare the early mask with the result from a run using unbounded propagation; if the early mask stays wrong, the bounded-memory pipeline has not preserved SAM2's correction behavior, contradicting the central accuracy claim.","supporting_citations":[{"cited_title":"Yolov8: A novel object detection algorithm with enhanced performance and robustness","cited_arxiv_id":null,"evidence_quote":"The YOLOv8 detection model supplies the automatic bounding-box prompts that replace human condition prompts."},{"cited_title":"Issue comment on sam2 github repository: Reuse the memory bank for another video","cited_arxiv_id":null,"evidence_quote":"A thread in the SAM2 codebase suggesting that a memory bank built for one video can be reused for another, giving rise to the preload memory bank feature."},{"cited_title":"Issue comment on sam2 github repository: Are there any method for reducing gpu memory over- head? https://github.com/facebookresearch/sam2/issues/196#issuecomment-2286352777, 2024","cited_arxiv_id":null,"evidence_quote":"A thread on reducing GPU memory overhead that inspired the release_old_frames and cache-clearing strategy."},{"cited_title":"Issue comment on sam2 github repository: Are there any method for reducing gpu memory over- head? https://github.com/facebookresearch/sam2/issues/196#issuecomment-2475114783, 2024","cited_arxiv_id":null,"evidence_quote":"A thread suggesting FP16 storage for video frames, used to reduce per-frame memory by roughly 0.007 GB."}],"review_version":1}