{"id":"a9001d72-872f-4af9-85c5-c5004f38925e","arxiv_id":"2412.04434","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"TROY-VIS makes open-vocabulary video instance segmentation run in real time (around 25 FPS) while keeping accuracy comparable to or better than slower state-of-the-art models.","lead":"This paper presents TROY-VIS, a video segmentation system that can recognize and track objects from any text description at about 25 frames per second, roughly 20 times faster than a strong baseline. The speed comes from three tricks: caching text embeddings, simplifying the attention mechanism, and reusing predictions from key frames on the frames in between.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper reports contradictory FPS figures for the final model (25 FPS in the abstract and Table 1 versus 20.9 FPS in Table 2), and the lower value falls below the paper's own 24 FPS real-time threshold.","rationale":"The reader's verdict is CONDITIONAL, and my analysis supports keeping that verdict rather than moving to accept or reject. The reader's weakest assumption focused on temporal consistency and kernel interpolation; that is a genuine risk, especially because the paper does not ablate the interpolation variant or analyze fast-motion failures. However, I see an even more immediate load-bearing problem in the speed measurement itself. The abstract and Table 1 give 40 ms total latency, which is 25 FPS, while Table 2 reports 20.9 FPS for the same final model. Since the paper's own real-time threshold is 24 FPS, this is not a cosmetic discrepancy: it determines whether the central novelty claim is true. The paper also does not state whether Table 2 FPS includes end-to-end overhead or how it relates to the Table 1 component latencies. This is a concrete, checkable issue, unlike the temporal-consistency concern, which is more of a robustness question and would require new failure-case experiments. My recommendation is therefore to keep the CONDITIONAL verdict, with the condition that the authors clarify and verify the FPS measurement and release code so the number can be reproduced. I am not alleging any misrepresentation; the inconsistency may come from different measurement conventions, but the burden is on the paper to make the central quantitative claim unambiguous.","tokens_in":13927,"tokens_out":6522,"duration_ms":63901,"concrete_test":"Obtain the released code and checkpoint, then measure end-to-end FPS on the LV-VIS validation set under the protocol stated in Table 1: batch size 1 on an A100 GPU, input short side resized to 480, including all preprocessing, postprocessing, and per-frame overhead. Report both total wall-clock FPS and per-component latency. If the measured FPS is at least 24, the real-time claim stands; if it is below 24, the abstract's 25 FPS and the 'real-time' characterization must be revised, and the comparison with GLEE-Lite should be re-run under the identical protocol. A secondary check should compare F=3 with F=1 to verify the kernel-interpolation speed contribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that TROY-VIS is the first real-time open-vocabulary video instance segmentation model, running 20x faster than GLEE-Lite. This claim rests entirely on the reported frame rate. The paper contains two incompatible numbers for the same final model: the abstract and Table 1 (40 ms total latency) imply 25 FPS, while Table 2 lists 20.9 FPS. The paper defines real-time as 24 FPS, so the two numbers land on opposite sides of the threshold. If 20.9 FPS is the correct end-to-end throughput (47.8 ms per frame), TROY-VIS is not real-time by the paper's own definition, and the speedup versus GLEE-Lite is about 16x rather than 20x. The text does not specify how the Table 2 FPS was measured relative to the Table 1 latency, and code and models are not yet released, so the discrepancy cannot be resolved from the manuscript. Because the headline novelty is precisely the real-time claim, this unresolved inconsistency is the most load-bearing concern: the speed claim is either true, false, or undefined depending on which number is the measured one.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TROY-VIS, a video instance segmentation architecture designed for open-vocabulary video instance segmentation (OV-VIS) at real-time speed. It identifies three computational bottlenecks in the GLEE-Lite baseline—text encoder, feature enhancer, and instance decoder—and introduces three corresponding techniques: a decoupled attention feature enhancer, a Flash Embedding Memory for caching text embeddings, and a kernel interpolation mechanism that propagates instance kernels from key frames to non-key frames. The authors report that TROY-VIS runs at 25 FPS on an A100 GPU, about 20x faster than GLEE-Lite, while achieving competitive or better accuracy on the BURST and LV-VIS benchmarks. The experiments include component-wise latency analysis (Table 1), zero-shot comparison with prior methods (Table 2), and an incremental ablation study (Table 3).","tokens_in":14099,"tokens_out":4724,"duration_ms":50291,"significance":"If the reported speed and accuracy numbers hold, this would be a practically valuable contribution: the first open-vocabulary video instance segmentation model that runs in real time, with strong zero-shot accuracy on two large-vocabulary benchmarks. The paper benefits from a clear component-wise latency breakdown, an incremental ablation design, and zero-shot evaluation on two external benchmarks, which makes the empirical claims easy to inspect. However, the central 'real-time' claim currently rests on internally inconsistent FPS numbers: the abstract and Table 1 imply 25 FPS, while Table 2 reports 20.9 FPS, which is below the paper's own 24 FPS real-time threshold. This inconsistency must be resolved before the headline claim can be accepted.","major_comments":[{"comment":"The FPS figures for the final model are mutually inconsistent. The abstract and the efficiency paragraph in Section 4.3 state 25 FPS (40 ms per frame), while Table 2 lists TROY-VIS at 20.9 FPS (47.8 ms per frame). Since Section 1 defines real-time as 24 FPS, the two numbers fall on opposite sides of the threshold. The manuscript does not explain whether the Table 2 FPS is an end-to-end measurement that includes text embedding retrieval, kernel interpolation, and post-processing, or whether the 40 ms in Table 1 is a component-only forward pass. This is load-bearing because the headline 'first real-time OV-VIS model' and the '20x faster than GLEE-Lite' claim are both direct consequences of the speed number; the authors need to report one consistent, clearly defined measurement protocol.","section":"Abstract; §4.3, Table 2; §1, Table 1"},{"comment":"The interpolation variant used in the final model is never identified. Section 3.3 presents three options: linear, nearest neighbor, and causal nearest neighbor. Neither Table 2 nor Table 3 states which one is used in the final TROY-VIS model. This matters because linear and bi-directional nearest-neighbor interpolation access future key frames, which is incompatible with the claimed online/real-time applicability; only causal nearest neighbor supports that claim. The paper also provides no ablation over the interpolation variant or over the key-frame interval F=3, even though F directly controls the speed/accuracy trade-off of the kernel interpolation mechanism.","section":"§3.3, Table 3"},{"comment":"Kernel interpolation assumes that elements of K(T) and K(T+3) correspond to the same object instances, but the paper does not describe how this correspondence is established. The instance decoder runs independently on each key frame with N unordered object queries; without a matching step, linearly combining kernel vectors across key frames mixes different instances, and even causal nearest-neighbor copying can attach a stale kernel to the wrong object after a reordering of queries. The authors need to specify the matching mechanism, or show experimentally that query order is stable enough for direct interpolation.","section":"§3.3, Eq. (1)"}],"minor_comments":[{"comment":"The final row of Table 3 reports AP=15.7 under the lightweight 100K-iteration training setting, whereas Table 2 reports AP=20.9 for TROY-VIS under the full 300K-iteration setting described in Section 3.4. The text should state this difference explicitly so that readers do not directly compare the two tables as if they reflect the same model.","section":"§4.4, Table 3"},{"comment":"For unseen categories encountered in the wild, the paper proposes averaging the embeddings of K nearest neighbors from the memory. This heuristic is plausible but is not validated in any experiment; a sentence noting this limitation, or a small analysis, would improve the presentation.","section":"§3.2"},{"comment":"The paper says the code and model will be released, but no code is currently available. Given that the speed claims depend on exact measurement conditions, the authors should release timing scripts or a precise measurement protocol at revision time.","section":"Abstract; §4.1"},{"comment":"Figure 1 states that 'TROY-VIS is the only method that runs in real-time.' This statement is only true if the FPS value is the 25 FPS figure from the abstract and Table 1; if Table 2's 20.9 FPS is the correct end-to-end value, this claim would be false under the paper's own 24 FPS definition. The figure and table should be made consistent.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"This is a solid systems paper with a clear ablation structure and useful zero-shot benchmark results. The only serious barrier is the unresolved FPS contradiction, which is central to the paper's main claim. I would ask the authors to provide a single, unambiguous timing protocol, correct the numbers across the abstract, tables, and figures, and clarify which interpolation variant is used and how instance correspondence is handled. If the corrected speed is at or above 24 FPS, the paper is likely acceptable; if it is below, the 'real-time' claim and the '20x faster' statement need to be revised."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the good news: TROY-VIS is a genuine systems-level advance. It takes GLEE's open-vocabulary VIS pipeline and makes it fast enough to be plausible for robotics and AR. The three components—decoupled attention, Flash Embedding Memory, and kernel interpolation—are each adapted from prior work (MobileInst, OmniDet-Turbo, depthwise separable convolutions), and the paper says so honestly. The ablations in Table 3 are clean and tell a coherent story: each component trades a bit of accuracy for a lot of speed, and the final combination lands at a good operating point. Evaluating zero-shot on BURST and LV-VIS is the right call, and the kernel interpolation trick being training-free is genuinely nice.\n\nThe soft spot is the speed number itself, and it's load-bearing. The abstract and Table 1 say 25 FPS (40 ms per frame); Table 2 lists 20.9 FPS for the final model. The paper defines real-time as 24 FPS, so these two numbers sit on opposite sides of the threshold. If 20.9 FPS is the correct end-to-end throughput, the 20x speedup over GLEE-Lite becomes about 16x and the real-time claim fails by the paper's own definition. The text never explains how the Table 2 FPS was measured relative to the Table 1 latency. Code and models aren't released, so this can't be resolved from the manuscript. This must be fixed before the headline claim is credible.\n\nSecond, the comparison to GLEE-Lite is uncontrolled. Flash Embedding Memory fixes a repeated text-encoder computation that is arguably a baseline inefficiency, and the accuracy gain mostly comes from the stronger EVA-02-CLIP-L text encoder and EfficientViT backbone, not the novel modules. The ablations actually show this: adding FEM+EVA gives +3.3 AP, then the layer reductions and kernel interpolation drop it back. That's fine, but the 'comparable or better accuracy' framing should acknowledge where the gains come from.\n\nMinor: F=3 is never ablated, the interpolation variants aren't compared in the final model, and there's no failure analysis for fast motion or occlusion under kernel interpolation. The BURST numbers don't collapse, so I consider this a moderate concern, not a fatal one.\n\nOverall: this is a solid engineering paper that deserves a serious referee. The central claim is important and the experiments are mostly well executed; the FPS inconsistency is addressable. I'd send it out, but require the authors to report one consistent end-to-end FPS measurement (ideally with code release) and to discuss the controlled comparison.","headline":"A capable engineering contribution to real-time OV-VIS, but the paper's own FPS numbers contradict each other and the lower one misses its real-time bar.","tokens_in":14749,"tokens_out":2798,"would_cite":false,"duration_ms":106890,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TROY-VIS segments arbitrary objects in video at 25 FPS, 20x faster than the prior best, with equal or better accuracy.","keywords":["open-vocabulary video instance segmentation","real-time inference","kernel interpolation","flash embedding memory","decoupled attention","temporal consistency","DETR-style decoder"],"falsifier":"Take a video benchmark with fast-moving or frequently occluded objects, run TROY-VIS with key frames spaced three frames apart, and compare tracking accuracy (HOTA or mask IoU) against running the instance decoder on every frame. If the gap in accuracy is much larger than the paper's reported 0.4 AP drop on LV-VIS, the temporal-consistency assumption is the limiting factor and the claim that real-time speed costs no accuracy would not generalize.","tokens_in":13638,"feed_emoji":"🎥","tokens_out":7851,"duration_ms":68440,"temperature":0.7,"pith_summary":"Open-vocabulary video instance segmentation (OV-VIS) has been accurate but too slow for real-world use: the lightweight GLEE-Lite foundation model processes about 1.25 frames per second on an A100 GPU. This paper argues that OV-VIS can run in real time without giving up accuracy, and introduces TROY-VIS, a model that runs at 25 FPS on the same hardware while matching or exceeding GLEE-Lite on the BURST and LV-VIS benchmarks. The route is to attack three computational bottlenecks: cross-modal attention, repeated text-embedding computation, and the per-frame instance decoder. If the claim holds, the practical obstacle to deploying open-vocabulary video understanding in robotics and augmented reality is removed.","feed_headline":"Video object segmentation hits real-time at 25 FPS","feed_subtitle":"Runs 20x faster than the previous best method, with matching or better accuracy on two benchmarks.","key_machinery":"The argument is carried by three mechanisms plus the instance-kernel representation they serve. Instance kernels are per-object weight arrays that are convolved with a downsampled pixel embedding map to produce masks, and the expensive part is producing those kernels with a 9-layer transformer decoder. The Decoupled Attention Feature Enhancer splits the original modality-scale hybrid cross-attention into a modality-attention between text and the lowest-resolution visual tokens and a scale-attention supplied by the existing deformable encoder, which relates points across feature scales, cutting the attention cost by a factor of roughly 85. Flash Embedding Memory stores category-name to text-embedding pairs so retrieval is O(1) instead of re-running the text encoder every frame, which also makes a stronger EVA-02-CLIP-L text encoder affordable. Kernel Interpolation assumes temporal consistency between frames: accurate kernels are computed on key frames spaced F=3 frames apart, and non-key frames use proxy kernels obtained by linear, nearest-neighbor, or causal nearest-neighbor interpolation; the causal variant keeps this compatible with online applications. The full model uses an EfficientViT-L2 backbone and only 3 encoder and 3 decoder layers.","core_discovery":"TROY-VIS is presented as the first real-time open-vocabulary video instance segmentation model. On LV-VIS it reaches 20.9 AP overall, 23.4 AP on base categories and 19.1 AP on novel categories, while GLEE-Lite reaches 19.6, 22.1 and 17.7; on BURST it reaches HOTA 23.9 on all categories and 42.3 on common categories, against 22.6 and 36.4 for GLEE-Lite. The speed gain to 40 ms per frame (25 FPS) comes from three changes: decoupling the feature enhancer's modality and scale attention, caching text embeddings in a Flash Embedding Memory so categories are encoded once, and interpolating instance kernels between key frames so the heavy decoder runs only one frame in three. The paper reports that these changes cut per-frame latency from 805 ms to 40 ms with no net loss of accuracy.","pith_inferences":["A direct test the paper does not run is varying the key-frame interval F: if F=3 already relies on temporal continuity, then F=1 versus F=3 on fast-motion videos would expose how much of the speed gain depends on static scenes.","The three interpolation variants are described but the final model's choice is not ablated; comparing linear, nearest-neighbor, and causal nearest-neighbor on the same benchmark would clarify which strategy the reported gains come from.","Flash Embedding Memory's nearest-neighbor fallback for unseen categories is likely to work better for categories near the existing vocabulary; rare or abstract categories may need a different retrieval rule, an extension the paper leaves implicit.","The same decoupled-attention and kernel-interpolation pattern could transfer to other DETR-style video perception tasks, such as open-vocabulary detection or panoptic segmentation, though the paper does not claim this."],"forward_implications":["Real-time open-vocabulary video understanding becomes possible on a single A100 GPU, the paper's stated enabler for mobile robotics and augmented reality.","The text-encoder bottleneck disappears on fixed-vocabulary benchmarks: category embeddings are computed once and reused, so larger text encoders can be swapped in at no inference cost.","Because kernel interpolation is applied only at inference, the model can be trained primarily on image-level annotations and does not require video data for the interpolation itself.","On both evaluated benchmarks, TROY-VIS sets the best accuracy numbers among efficient OV-VIS methods while being an order of magnitude faster than GLEE-Lite.","Reducing the decoder from 9 to 3 layers is shown to trade 1.2 LV-VIS AP for a 35 ms latency saving, and kernel interpolation adds a further 0.4 AP drop while raising speed from 22.2 to 31.3 FPS."],"supporting_citations":[{"why":"Supplies the GLEE/GLEE-Lite baseline whose per-component latency, architecture, and benchmark numbers TROY-VIS is designed against.","marker":"[40]"},{"why":"Defines the BURST benchmark, whose HOTA and mAP metrics are one of the two zero-shot evaluations.","marker":"[3]"},{"why":"Defines the LV-VIS benchmark with 1196 categories, whose AP is the main accuracy measure, and introduces the OV2Seg baseline.","marker":"[38]"},{"why":"Supplies the EfficientViT-L2 vision backbone chosen for its accuracy-speed balance.","marker":"[5]"},{"why":"Supplies the EVA-02-CLIP-L text encoder that Flash Embedding Memory makes affordable.","marker":"[12]"},{"why":"Provides the Mask DINO decoder design that the instance decoder is built from and whose per-frame cost kernel interpolation reduces.","marker":"[21]"}],"fun_headline_variants":["TROY-VIS: Open-vocabulary video segmentation at 25 FPS","Open-vocabulary video segmentation hits real-time at 25 FPS","20x faster video instance segmentation without accuracy loss","Real-time open-vocabulary segmentation: TROY-VIS runs 25 FPS"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that objects change little between key frames spaced three frames apart, so kernels computed on an earlier frame still match the current frame; fast motion, occlusion, or appearance change inside that gap would make the reused predictions stale.","fun_headline_variants_meta":{"raw":{"variants":["TROY-VIS: Open-vocabulary video segmentation at 25 FPS","Open-vocabulary video segmentation hits real-time at 25 FPS","20x faster video instance segmentation without accuracy loss","Real-time open-vocabulary segmentation: TROY-VIS runs 25 FPS"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000796,"raw_usage":{"total_tokens":3505,"prompt_tokens":947,"completion_tokens":2558,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":2489}},"tokens_in":563,"tokens_out":2558,"duration_ms":17098,"temperature":1.0,"reasoning_tokens":2489,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:23:22.574753+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a video benchmark with fast-moving or frequently occluded objects, run TROY-VIS with key frames spaced three frames apart, and compare tracking accuracy (HOTA or mask IoU) against running the instance decoder on every frame. If the gap in accuracy is much larger than the paper's reported 0.4 AP drop on LV-VIS, the temporal-consistency assumption is the limiting factor and the claim that real-time speed costs no accuracy would not generalize.","supporting_citations":[{"cited_title":"General object foundation model for images and videos at scale","cited_arxiv_id":null,"evidence_quote":"Supplies the GLEE/GLEE-Lite baseline whose per-component latency, architecture, and benchmark numbers TROY-VIS is designed against."},{"cited_title":"Burst: A benchmark for unifying object recognition, segmentation and tracking in video","cited_arxiv_id":null,"evidence_quote":"Defines the BURST benchmark, whose HOTA and mAP metrics are one of the two zero-shot evaluations."},{"cited_title":"Towards open-vocabulary video instance segmentation","cited_arxiv_id":null,"evidence_quote":"Defines the LV-VIS benchmark with 1196 categories, whose AP is the main accuracy measure, and introduces the OV2Seg baseline."},{"cited_title":"Efficientvit: Lightweight multi-scale attention for high- resolution dense prediction","cited_arxiv_id":null,"evidence_quote":"Supplies the EfficientViT-L2 vision backbone chosen for its accuracy-speed balance."},{"cited_title":"Mask dino: Towards a unified transformer-based framework for object detection and segmentation","cited_arxiv_id":null,"evidence_quote":"Provides the Mask DINO decoder design that the instance decoder is built from and whose per-frame cost kernel interpolation reduces."}],"review_version":1}