{"id":"ff5f982d-04ab-4a02-bb3e-8d3f470a4bf1","arxiv_id":"2411.13183","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Click-based initialization with a text-guided box refiner (GCR) can initialize single-object trackers at 31 FPS with accuracy close to precise box annotation.","lead":"ClickTrack replaces the precise bounding-box initialization of single-object trackers with a single click on the target, plus an optional text category to disambiguate parts such as a license plate versus the whole car. The authors report that their Guided Click Refiner produces boxes that let standard trackers match most of the accuracy of hand-drawn initialization frames.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Click-simulation mismatch is the load-bearing concern: GCR is trained and evaluated with uniform ellipse-sampled points (Sec. 4.2, Table 7), so its reported stability may not transfer to real operator clicks.","rationale":"The reader's weakest assumption and my load-bearing concern coincide: the paper's entire empirical case depends on a synthetic click distribution that may not match how operators actually click. This is not a minor methodological detail; it is the bridge between the reported benchmark numbers and the claimed real-time interactive paradigm. The paper does provide a robustness analysis (Table 7) and a simulation study (Table 8), but both use the same uniform-ellipse distribution as training, so they only establish insensitivity to perturbations within that distribution, not to distribution shift. The text-input path is also only simulated with oracle category labels, not with real operator-provided text, which further weakens the interactive claim. I considered whether the training-on-the-target-benchmark protocol (GCR trained on LaSOT/GOT-10k training sets) is a more load-bearing flaw; it does undermine the fairness of comparisons to frozen zero-shot baselines like SAM and GroundingDINO, but the central claim of feasibility does not strictly depend on beating those baselines. The click-distribution mismatch, by contrast, directly threatens the conclusion that ClickTrack works in the intended deployment setting. The reader's CONDITIONAL verdict already accounts for this weakness, so my analysis does not move the verdict; it sharpens the required condition: a user study or at least a realistic click-distribution test is necessary before accepting the interactive robustness claim. I therefore recommend UNCHANGED, with the condition explicitly tied to this validation step.","tokens_in":17831,"tokens_out":5609,"duration_ms":61021,"concrete_test":"Conduct a small user study: recruit 10 operators to click once on the target object in the first frame of a random subset of 100 LaSOT test videos, using the same category prompts as the Point+Text condition. Run GCR+STARK on these clicks and compare the success rate with the reported 65.0 from uniform ellipse sampling. If the user-click success rate drops below 62.0 (or by more than 3 points), the synthetic simulation overstates real-world robustness; if it stays within 3 points, the concern is mitigated. To make the test reproducible, release the collected click coordinates along with the paper's code.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that b = T(S, R(p0)) makes click-based initialization practical, with 65.0 success on LaSOT and 63.2 mAO on GOT-10k. The evidence rests on an assumed click distribution: during training and robustness testing, points are sampled uniformly inside an ellipse with semi-axes one-fourth of the ground-truth box width and height (Sec. 4.2; Table 7; Table 8). This distribution is used both for training GCR and for evaluating it, so the reported robustness to click position is essentially an interpolation result. Real operator clicks are not uniformly distributed over an object; they typically cluster near the center or on semantically meaningful parts, and occasional clicks land on boundaries or ambiguous regions (the paper itself motivates text input with the license-plate example in Fig. 3). When text is used in the experiments, the category labels come from the dataset or from manual annotation (DAVIS/YT-VOS), not from real operator speech or typing. Consequently, the quantitative claim that ClickTrack provides 'stable performance in real-time interactive scenarios' is not yet grounded in actual human interaction data. This is the single most load-bearing concern because the paradigm's value proposition is precisely its usability in real-time interactive setups. The paper's synthetic robustness study cannot rule out a distribution shift that could substantially degrade performance under realistic clicks.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ClickTrack, a new paradigm for initializing single-object trackers in which the operator provides a single click (with optional category text) and a learned point-to-box refiner, GCR, converts the click into an initial bounding box for an off-the-shelf tracker. GCR consists of a Guided Convolution that fuses visual RoI features with a text or learnable guiding feature, a Prototype Selection module that picks an anchor prototype, and an Iterative Refinement module that cascades refinements. Experiments combine GCR with STARK on LaSOT and GOT-10k, reporting 65.0 success on LaSOT with point+text and 63.2 mAO on GOT-10k with point only; ablations analyze the modules, number of stages, anchor settings, and robustness to click position. The paper also extends the GCR head to SAM (GCR-SAM) and reports improved box/mask IoU on COCO and improved VOS initialization on DAVIS 2017 and YT-VOS 2018.","tokens_in":18092,"tokens_out":8361,"duration_ms":76871,"significance":"If the reported results held under realistic interactive conditions, ClickTrack would be a valuable contribution: it replaces precise box drawing with a single click, which is faster and more stable, and the optional text input addresses ambiguity. The GOT-10k comparison, where all methods are trained on the same split, provides some evidence that GCR improves over simpler point-to-box baselines, and the paper ships extensive ablations and a transfer study to SAM. A notable strength is that GCR-SAM is evaluated on DAVIS and YT-VOS without using their training data. However, the significance is tempered by the fact that the robustness evaluation uses the same synthetic click distribution as training, the LaSOT comparison is not controlled for in-domain training, and the GCR-SAM comparison is confounded by the additional text input. The headline claims of real-time interactive stability are therefore not yet fully established.","major_comments":[{"comment":"The training procedure samples random points uniformly inside an ellipse with semi-axes one-fourth of the ground-truth box width and height, and the robustness analysis in Table 7 evaluates the same distribution by re-sampling points at training time. This makes the robustness result an interpolation result under the training distribution rather than evidence about realistic operator clicks, which are known to cluster near meaningful parts or centers and can land on boundaries. Since the paper's value proposition is specifically real-time interactive scenarios, I consider this mismatch load-bearing. Please add at least one of the following: a user study with human clicks, an evaluation on a dataset containing human click annotations, or a sensitivity analysis with alternative click distributions (e.g., center-biased Gaussian, boundary clicks, part-level clicks) that are not used in training.","section":"Section 4.2 / Table 7"},{"comment":"The headline comparison on LaSOT is not apples-to-apples: GCR is trained on the LaSOT training split, while the SAM, OVSAM, and GroundingDINO baselines are used off-the-shelf and are not fine-tuned on LaSOT. The performance gap in Table 1 (e.g., 65.0 vs 60.1 for SAM-B in the Point setting) may therefore be due to in-domain training rather than to the proposed architecture. On GOT-10k all methods are trained on the same split, which supports the comparison there. For a clean LaSOT comparison, please either fine-tune the baselines on LaSOT, report GCR in a zero-shot or cross-dataset configuration, or explicitly state the training protocol of each method.","section":"Table 1 / Section 4.2"},{"comment":"The simulation experiments comparing box and point initialization do not specify how the 'Box' initialization boxes are generated. The reported success of 54.1 for Box is far below the known performance of precise-box STARK on LaSOT (typically above 60), indicating that the boxes are corrupted by an unspecified noise model. Without a definition of this model, the conclusion in Section 4.5 that 'point input provides higher accuracy for single-object tracking initialization' is uninterpretable. Please specify the box noise protocol and, ideally, compare against a precise-box upper bound.","section":"Table 8 / Section 4.5"},{"comment":"The GCR-SAM experiments compare GCR-SAM (which always receives a category text prompt) against SAM baselines that receive only the point prompt. The reported gains could therefore be attributed to the additional text modality rather than to the GCR head, and the claim that the GCR structure itself alleviates SAM's ambiguity is not isolated. Please report a point-only GCR-SAM variant (using the learnable proposal feature) alongside the text-conditioned variants, or otherwise control for the text input in the comparison.","section":"Section 5.3 / Tables 9-11"}],"minor_comments":[{"comment":"Table 1 contains the typos 'Dtector + Point' and 'Detector (top.1)', and Table 8 contains 'Norm-Pricise' instead of 'Norm-Precise'.","section":"Table 1 / Table 8"},{"comment":"The caption of Figure 8 reads 'The framework of TCR-SAM' but should be 'The framework of GCR-SAM'.","section":"Figure 8"},{"comment":"The sentence 'Each epoch randomly samples 64,000 images in the training dataset' is ambiguous; please clarify whether 64,000 is the number of images per epoch or per iteration.","section":"Section 4.2"},{"comment":"In Tables 10 and 11, the 'Upper Bound' row for UNINEXT on YT-VOS presents numbers that run together (e.g., '72.1 72.3 76.2 65.9 74.177.0 76.8 81.0 70.8 79.4'), which should be reformatted for readability.","section":"Tables 10-11"},{"comment":"The paper does not mention whether code or trained models will be released; providing them would support reproducibility of the reported numbers.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of Pattern Recognition and the central idea is appealing. The main concerns are evaluation-protocol issues rather than conceptual flaws: the click-simulation mismatch, the uncontrolled LaSOT comparison, the unspecified box-noise model in Table 8, and the text-input confound in the GCR-SAM experiments. I recommend major revision with emphasis on these points."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a competent applied tracking paper with a modest but real contribution. The idea is to replace the hand-drawn initial box with a click plus optional text category, via a refiner (GCR) that plugs into any tracker. That framing is sensible, and the GCR design – guided convolution, prototype selection, iterative refinement – is reasonable engineering. The strongest evidence is on GOT-10k, where all refiners are trained on the same data: GCR beats BFR and PFR by a clear margin (63.2 vs 56.3/61.5 mAO). That supports the central claim that the architecture helps, not just the general point-to-box idea.\n\nThe LaSOT comparison in Table 1 is less clean. GCR is fine-tuned on LaSOT training data, while the SAM/OVSAM/GroundingDino baselines are frozen pre-trained models. That is not a fair comparison, and the reported margins there (e.g., 65.0 vs 60.1) likely overstate the advantage. The paper calls it fair, which is wrong. Still, the GOT-10k result stands, and the ablation study on LaSOT is internally consistent.\n\nThe bigger soft spot is the click model. Training and evaluation sample points uniformly inside an ellipse with semi-axes one-quarter of the box. Real operator clicks are not uniform – they cluster near centers or meaningful parts, and some land on boundaries. The robustness study (Table 7) re-samples from the same distribution, so it only shows stability within that synthetic setting. Without a user study or at least a broader sampling test, the headline claim about 'real-time interactive scenarios' remains a promise. The paper also uses manually added category labels on DAVIS/YT-VOS, not operator-provided text.\n\nMinor: no code, no error bars, and the 'new paradigm' framing oversells what is a composition of known ideas (point-to-box regression plus a tracker). The paper does cite P2BNet and coarse point supervision, so the prior work is acknowledged.\n\nBottom line: the paper deserves a serious referee. The core evidence on GOT-10k is sound, the ablation is careful, and the GCR-SAM extension is a nice bonus. The unfair LaSOT comparison and the click-simulation gap are fixable in revision. I would send it to review with a request for a user study or a more realistic click model, and a reworked comparison on LaSOT.","headline":"ClickTrack cleanly packages a known point-to-box idea for SOT initialization and backs it with solid GOT-10k evidence, but the LaSOT comparison is unfair and the click simulation is too narrow to support the real-time interactive claim.","tokens_in":18680,"tokens_out":2402,"would_cite":true,"duration_ms":23408,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single click, optionally labeled with a category, can initialize a single-object tracker almost as well as a hand-drawn bounding box.","keywords":["single object tracking","click interaction","point initialization","Guided Click Refiner","bounding box regression","video object segmentation","text-guided refinement","real-time interactive tracking"],"falsifier":"Run a user study on LaSOT in which operators freely click on the target object and optionally speak a category, feed those genuine clicks through GCR, and compare the success rate with the synthetic-ellipse numbers (62.4 point-only, 65.0 with text); a substantial drop would show that the ellipse simulation does not model real clicking behavior.","tokens_in":1753,"feed_emoji":"🎯","tokens_out":1965,"duration_ms":62155,"temperature":0.7,"pith_summary":"Single-object trackers normally demand a precise bounding box around the target in the first frame, and even small annotation errors visibly degrade long-term accuracy. This paper proposes replacing that box with a single click, optionally accompanied by a one-word category when the target is ambiguous, and letting a learned refiner generate the box. The Guided Click Refiner (GCR) turns the click into the four edges of the operator's intended box, is tracker-agnostic, and runs fast enough for real-time interactive use. Combined with a transformer tracker, it reports a 65.0 success score on LaSOT and 63.2 average overlap on GOT-10k, close to the same tracker's performance with hand-drawn boxes. If this holds, pointing at an object becomes a sufficient initialization for robust tracking, and the same refiner also cleans up point-based segmentation with models like SAM.","feed_headline":"One click starts tracking almost as well as a hand-drawn box","feed_subtitle":"ClickTrack's refiner turns a click plus optional category into a box, matching precise-annotation accuracy on LaSOT and GOT-10k.","key_machinery":"The load-bearing object is the Guided Click Refiner (GCR), a point-to-box regressor built from three parts. Guided Convolution (GC) is the core: it takes an RoI feature and a guiding feature — either a CLIP text embedding of the object category or a learnable proposal feature — and uses the guiding feature to generate dynamic projection parameters that modulate the visual feature, so the text can steer which part of the object the box should cover. Prototype Selection (PS) places several anchor boxes around the click, uses GC to adjust them and to predict each adjusted anchor's IoU, and keeps the highest-scoring one as the initial regression region. Iterative Refinement (IR) then cascades several GC-like stages to narrow the box toward the target. The whole pipeline regresses four distances from the click to the box edges, which converts the point into a tracker-ready template.","core_discovery":"The paper's central claim is that single-object tracking can be reformulated as $b = \\mathcal{T}(\\mathcal{S}, \\mathcal{R}(p_0))$: given a tracker $\\mathcal{T}$, a video $\\mathcal{S}$, and an initial click $p_0$, a point-to-box regressor $\\mathcal{R}$ produces the initial template box, so no hand-drawn bounding box is required. The regression targets are the distances from the click to the four edges of the target box. To resolve the ambiguity that a bare click carries, $\\mathcal{R}$ is the Guided Click Refiner (GCR), which accepts an optional text or learnable guiding feature: a Guided Convolution fuses that feature into the visual RoI representation, a Prototype Selection module picks the most promising anchor region by predicted IoU, and an Iterative Refinement cascade sharpens the box. With this design and CLIP text features for category guidance, GCR plus STARK reaches 65.0 success and 68.3 precision on LaSOT using point-plus-text, and 62.4 success with point alone; on GOT-10k it reaches 63.2 mean average overlap. The same GCR structure, attached to SAM as a prompt generator, lifts COCO mask overlap from roughly 52 to 70 IoU, showing the ambiguity cure transfers beyond tracking.","pith_inferences":["The training-time assumption that clicks land uniformly inside an ellipse one-quarter the box width and height is a proxy for real operator behavior; genuine human clicks on boundaries, on occluded parts, or on background clutter may not follow that distribution, so a user study would be the natural next test.","Because the guiding feature can be a learnable vector as well as text, GCR could be extended to accept other cheap cues — a voice keyword, a finger gesture, or even a rough sketch — without changing the architecture.","The open-vocabulary behavior of the text path is inherited from CLIP, so the practical ceiling of the text guidance is set by the category vocabulary CLIP has seen; a category outside that vocabulary would likely fall back to the learnable-proposal behavior.","The same point-plus-optional-category pattern could be applied to other interactive vision tasks that currently require precise boxes or masks, such as interactive detection or multi-object tracking initialization."],"forward_implications":["Real-time interactive trackers can be initialized by a point click instead of a laborious box annotation, removing a major usability bottleneck in live systems.","Adding a one-word category (e.g., 'license plate' vs 'car') measurably removes ambiguity, raising LaSOT success from 62.4 to 65.0 when combined with text.","GCR is tracker-agnostic: it works with STARK, MixFormer, and PrDimp, so the click-to-box module can be bolted onto any existing single-object tracker.","The refiner transfers to segmentation: plugging GCR into SAM raises single-point mask IoU on COCO substantially and improves VOS initialization on DAVIS and YT-VOS.","GCR runs at 31 FPS and adds only about 0.03 seconds per video sequence, so the new initialization paradigm preserves real-time operation."],"supporting_citations":[{"why":"LaSOT supplies the main benchmark on which the success-rate comparisons between initialization methods are run.","marker":"[1]"},{"why":"GOT-10k supplies the second benchmark used to measure transfer of the click-initialized tracker.","marker":"[12]"},{"why":"STARK is the transformer tracker that consumes the GCR-generated boxes in the core experiments.","marker":"[13]"},{"why":"SAM is the segmentation model into which GCR is plugged for the GCR-SAM extension experiments.","marker":"[9]"},{"why":"CLIP provides the text features that guide the GCR regression when category information is used.","marker":"[36]"},{"why":"GroundingDINO is a point-plus-text initialization baseline that GCR is compared against.","marker":"[48]"},{"why":"OVSAM is an open-vocabulary SAM baseline compared for both point and point-plus-text initialization.","marker":"[49]"},{"why":"Dynamic Convolution inspires the design of Guided Convolution, where the interaction parameters are generated from guiding information.","marker":"[44]"},{"why":"Faster R-CNN is one of the detectors whose structure motivates the two naive point-based baseline refiners.","marker":"[41]"},{"why":"FCOS is the other detector structure used to motivate the naive point-based baseline refiners BFR and PFR.","marker":"[42]"}],"fun_headline_variants":["Click once, track any object in real time without boxes","One click becomes a precise box for live object tracking","Point plus text: a click becomes a box for real-time tracking","No box needed: click to track with optional text in real time","Click-based tracking: no hand-drawn boxes, just a point"],"cache_read_input_tokens":20736,"weakest_assumption_plain":"The reported stability rests on the assumption that uniformly sampling points inside an ellipse whose semi-axes are one-fourth of the ground-truth box width and height reproduces how real operators actually click; there is no user study checking this, and on video-segmentation benchmarks the category labels were added manually rather than spoken by an operator.","fun_headline_variants_meta":{"raw":{"variants":["Click once, track any object in real time without boxes","One click becomes a precise box for live object tracking","Point plus text: a click becomes a box for real-time tracking","No box needed: click to track with optional text in real time","Click-based tracking: no hand-drawn boxes, just a point"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001226,"raw_usage":{"total_tokens":5054,"prompt_tokens":978,"completion_tokens":4076,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":3990}},"tokens_in":594,"tokens_out":4076,"duration_ms":28271,"temperature":1.0,"reasoning_tokens":3990,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:44:26.898912+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a user study on LaSOT in which operators freely click on the target object and optionally speak a category, feed those genuine clicks through GCR, and compare the success rate with the synthetic-ellipse numbers (62.4 point-only, 65.0 with text); a substantial drop would show that the ellipse simulation does not model real clicking behavior.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LaSOT supplies the main benchmark on which the success-rate comparisons between initialization methods are run."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"STARK is the transformer tracker that consumes the GCR-generated boxes in the core experiments."},{"cited_title":"Radford, J","cited_arxiv_id":null,"evidence_quote":"CLIP provides the text features that guide the GCR regression when category information is used."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"OVSAM is an open-vocabulary SAM baseline compared for both point and point-plus-text initialization."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Dynamic Convolution inspires the design of Guided Convolution, where the interaction parameters are generated from guiding information."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FCOS is the other detector structure used to motivate the naive point-based baseline refiners BFR and PFR."}],"review_version":1}