{"id":"7af2ed9d-877e-4361-b834-a39e1b2862b4","arxiv_id":"2501.11153","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Kinematics Adaptive Frame Recognition selects frames with significant tool motion, reducing training data 5x to 10x and modestly improving phase segmentation accuracy over uniform sampling.","lead":"This paper proposes KAFR, a method that selects a small subset of video frames for training surgical phase segmentation models by tracking how much the surgical tools move. The authors report cutting the frame count by 5x to 10x while slightly improving accuracy on two robotic surgery datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline accuracy gain is confounded by two-stream ensembling: Table 5 shows RGB-only KAFR (0.7438) is below the baseline (0.749), and the 0.7814 result uses 15% of frames, not the claimed tenfold reduction.","rationale":"The reader correctly identifies that the central claim is unsupported, and their rationale explicitly mentions the confound between the two-stream ensemble and the single-stream baseline, as well as the best-result selection from a large grid. However, the reader's formal 'weakest_assumption' focuses on the kinematic proxy (frames with little tool motion are informationally redundant), which is a different, less decisive issue: even if that proxy is imperfect, the method could still be beneficial on average, and the paper itself acknowledges a failure case. The more load-bearing problem is that the paper's own numbers show the KAFR-selected RGB stream is worse than the baseline, so the reported accuracy gain does not come from frame selection at all. This is an internal inconsistency rather than a disagreement with field consensus, and it directly invalidates the headline claim. A controlled single-stream comparison, as specified in the concrete test, would settle whether any accuracy gain remains after removing the ensembling confound. If the single-stream KAFR result still exceeds the baseline with proper error bars, the method would have real value; if not, the paper's central claim fails. The reader's REJECT verdict is appropriate based on the evidence as presented, so no verdict adjustment is needed.","tokens_in":20737,"tokens_out":3983,"duration_ms":36989,"concrete_test":"Rerun the GJ comparison with matched architecture and data proportion: train a single-stream RGB X3D model using the prior work's exact hyperparameters on (a) the full training set and (b) the KAFR-selected 15% two-object acceleration subset, with at least three random seeds each, and compare validation accuracy with a paired significance test. If single-stream RGB KAFR on the subset does not exceed the full-set baseline, the headline improvement is fully explained by ensembling. Also report the actual retained-frame fraction for the 0.7814 configuration to verify the claimed tenfold reduction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim, that KAFR reduces dataset size while improving phase segmentation accuracy, is not supported by the reported experiments because the headline improvement is confounded by an architecture change. The best GJ result (accuracy 0.7814) comes from ensembling two X3D models, one on RGB frames and one on optical flow, trained on a KAFR-selected 15% subset; the baseline (0.749) is a single X3D model trained on all frames. Table 5 directly exposes this confound: for the same KAFR-selected 15% subset (two objects, acceleration), the RGB-only stream achieves 0.7438, which is below the 0.749 baseline. Thus the reported 4.32% relative accuracy increase is attributable to ensembling two independent models, not to the kinematic frame-selection mechanism. Moreover, the abstract and conclusion claim a 'tenfold reduction' in frames for this result, but Table 5 shows the best configuration uses 15% of frames, a reduction of roughly 6.7x; the only 10% result in Table 4 achieves 0.7684, which is not the headline number. No error bars, repeated seeds, or a proper control matching model capacity are provided, and the best result is selected from a large grid of configurations (object counts, frame percentages, velocity/acceleration, and stream choices), so selection effects cannot be ruled out. The new GJ dataset and public code are valuable contributions, but the evidence presented does not isolate KAFR's contribution.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Kinematics Adaptive Frame Recognition (KAFR), a frame-subsampling method for surgical phase segmentation. Frames are selected by tracking surgical tool centroids with YOLOv8 and thresholding a decreasing function of cumulative centroid displacement (Adaptive 1) or variation of velocity (Adaptive 2). Selected key frames are used to train X3D CNNs; in the final configuration two X3D models (RGB and optical flow) are ensembled. The authors introduce a newly annotated Gastrojejunostomy (GJ) dataset (42 videos, six phases plus idle) and reuse a Pancreaticojejunostomy (PJ) dataset, reporting a 4.32% relative accuracy gain with a claimed tenfold frame reduction on GJ and a 2.05% gain with fivefold reduction on PJ.","tokens_in":21074,"tokens_out":7525,"duration_ms":69127,"significance":"If the reported gains were attributable to KAFR, the paper would offer a practical way to reduce the computational cost of training on long surgical videos, and the new GJ dataset plus public code would be useful community resources. However, the experiments as reported do not isolate KAFR's effect: the headline result uses a two-stream ensemble whose individual RGB stream underperforms the single-stream baseline, and the 'tenfold' claim corresponds to a different configuration than the one producing the headline accuracy. The central claim is therefore currently unsupported, even though the underlying hypothesis is interesting and the dataset/code contributions are real.","major_comments":[{"comment":"The headline GJ result of 0.7814 in Table 5 is produced by ensembling two X3D models (RGB and optical flow) trained on a 15% KAFR subset, while the baseline of 0.749 is a single X3D model. The RGB-only KAFR stream in the same configuration reaches only 0.7438, below the baseline, and the optical-flow-only stream reaches 0.7291. The reported 4.32% gain is therefore confounded by the addition of a second model and input modality and cannot be attributed to KAFR. The same confound applies to PJ: the 0.8982 result in Section VI is explicitly obtained 'with the ensembling method,' whereas the baseline 0.8801 is a single-stream X3D model from prior work.","section":"V-D, Table 5; Section VI"},{"comment":"The abstract and conclusion claim a 'tenfold reduction' in frames accompanies the accuracy 0.7814, but the configuration producing 0.7814 uses 15% of the frames (Table 5), which is roughly a 6.7x reduction, not tenfold. The only 10% configuration in Table 4 (Four Objects, Velocity) gives 0.7684, a 2.53% gain, not the headline 4.32% gain. The paper appears to combine the best accuracy from one configuration with the largest reduction from another.","section":"Abstract/Conclusion; Tables 4 and 5"},{"comment":"The evaluation selects the reported result from a large grid of configurations. The threshold is 'determined based on the desired number of extracted frames' (Section IV-C), so the retained frame fraction is a free parameter; the object count (one/two/four), the kinematic signal (velocity vs acceleration), and the stream combination are all varied, and the best accuracy is reported without error bars or repeated-seed variance. With 21 configurations in Table 4 alone, plus the two-stream configurations in Table 5 and the PJ results in Section VI, selection effects are a serious concern. In addition, the GJ baseline of 0.749 is not fully specified; the text refers to 'settings described in prior work' rather than reporting a GJ-specific run with the same protocol.","section":"IV-C, V-A, V-C"},{"comment":"The authors themselves document a case where KAFR removes informative frames: in the Enterotomy phase, the untracked right-hand tool performed the critical cutting, so the tracked left-hand tool showed little motion and KAFR produced a large gap in the selected frames. This concession directly undermines the paper's core assumption that frames with little tracked-tool motion are redundant, and the paper provides no phase-level or per-case analysis quantifying how often this failure occurs. At minimum, robustness to this failure mode needs to be demonstrated before the method can be recommended as generally applicable.","section":"V-D, Figure 13"}],"minor_comments":[{"comment":"The notation in Equations (2) and (5) is inconsistent: D is defined with a sum over s in S, but the term ||s(x_i) - s(x_k)|| treats s as a point index, and the subset s mentioned in the text plays no role. Please rewrite these definitions with clear per-tool centroid notation.","section":"Equations (2) and (5)"},{"comment":"The entry for Phase 2 of GJ05262020PH reads '1 4.80%', presumably '14.80%'.","section":"Table 6"},{"comment":"Section IV-D defines accuracy and F1 changes as relative changes, but the abstract's '4.32%' and '0.16%' could easily be read as absolute percentage points; please state this explicitly wherever such gains are reported.","section":"IV-D"},{"comment":"The state-of-the-art comparison in Table 7 lists results on Cholec80, Cholec51, CATARACTS, and M2cai16, which are different datasets and evaluation protocols; this is not a controlled comparison with the proposed method and should be relabeled as context rather than a competitive evaluation.","section":"VII, Table 7"},{"comment":"The runtime example in Section VII (approximately 19'15'' per epoch on full data versus 5'24'' on 10% of the data) is not a tenfold speedup and is closer to 3.5x; please report measured wall-clock times instead of extrapolating linearly from the data fraction.","section":"Section VII, runtime discussion"}],"recommendation":"reject","confidential_remarks":"The new GJ dataset and the public code are valuable, but the central claim is not supported by the current experimental design. A revision would require substantially new controlled experiments—at minimum, a single-stream RGB KAFR model compared against a single-stream RGB baseline, a two-stream baseline trained on all frames, and repeated-seed variance reporting. Without those, the reported gains cannot be interpreted as evidence for KAFR."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the kinematic keyframe idea is sensible and the new GJ dataset is a real asset, but the headline accuracy gain is not supported by the reported experiments. The best GJ number (0.7814) comes from ensembling two X3D models (RGB + optical flow) trained on a 15% KAFR subset; the baseline is a single-stream model on all frames. Table 5 shows the RGB-only stream on that same subset gets 0.7438, below the 0.749 baseline. So the reported 4.32% improvement likely reflects the extra optical-flow stream, not the frame selection. The abstract also says \"tenfold reduction\" while the best configuration keeps 15% of frames (roughly 6.7x); the 10% result is 0.7684, not the headline.\n\nWhat's genuinely useful: the KAFR idea itself—using tracked tool centroid displacement, velocity, or acceleration to drop redundant frames—is a reasonable heuristic and, as far as I know, not in the cited keyframe literature. The new GJ dataset (42 videos, six phases, annotated at two centers) plus public code is a contribution the community can build on. The authors also explicitly document a failure case (Enterotomy, where the untracked right-hand tool did the cutting) and the stationary-camera assumption—that honesty is to their credit.\n\nThe soft spots are the ones above, plus: no error bars or multiple seeds; results are picked from a large grid (object count, frame percentage, velocity vs acceleration, stream choice); and the PJ comparison has the same architecture-change issue. A properly matched comparison—same single-stream X3D, same training length, KAFR versus uniform sampling, several seeds—is needed to isolate the effect. That said, the math is straightforward and there's no circularity; the method is evaluated on held-out validation, not the training set.\n\nWho should read this: anyone working on surgical phase segmentation or efficient video training. It's not a paper to cite for the accuracy claim as it stands, but the dataset and the failure analysis are worth knowing. I'd send it to review, not desk-reject, because the underlying question is real and the confound is fixable with a cleaner experiment.","headline":"The core claim is confounded by a two-stream ensemble; the RGB-only KAFR result is below baseline, so the paper needs a controlled comparison before the headline can be believed.","tokens_in":21660,"tokens_out":1880,"would_cite":false,"duration_ms":16849,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Surgical phase segmentation can be trained on one-tenth of the frames and still gain accuracy when frame selection is driven by tracked surgical tool motion.","keywords":["Kinematics Adaptive Frame Recognition","Surgical phase segmentation","Tool tracking","Key frame extraction","Convolutional Neural Networks","Deep learning","Surgical video analysis","YOLOv8"],"falsifier":"Run the same two-stream X3D setup on a surgical video where the phase-defining action is performed by an untracked tool or by camera motion, with the Enterotomy example in Figure 13 as a natural candidate; if KAFR selects fewer frames in those segments than a uniform sampler at equal retention and its accuracy drops to or below the uniform-sampler level, the assumption that low centroid motion equals redundancy is falsified.","tokens_in":20500,"feed_emoji":"🩺","tokens_out":7964,"duration_ms":75399,"temperature":0.7,"pith_summary":"The paper sets out to show that surgical phase segmentation can be made cheaper and more accurate by deleting frames that look redundant from the standpoint of instrument motion. It introduces KAFR, a frame-selection rule that keeps only frames where the tracked centroids of surgical tool parts move enough, using accumulated displacement in Adaptive 1 or accumulated velocity change in Adaptive 2, and trains X3D CNNs on that subset. On a newly annotated Gastrojejunostomy dataset, the method reports a tenfold reduction in training frames with a 4.32% relative accuracy gain (0.749 to 0.7814); on a Pancreaticojejunostomy dataset, it reports a fivefold reduction with a 2.05% relative accuracy gain (0.8801 to 0.8982). A sympathetic reader would care because, if true, frame reduction is not a trade-off against accuracy but a route to better accuracy with lower storage and computation costs.","feed_headline":"Tracking tools slashes surgical video frames 10x and boosts accuracy","feed_subtitle":"Instrument motion picks the frames worth training on, and phase accuracy still rises with one-tenth the data.","key_machinery":"The machinery is the KAFR selection rule and its two variants. Adaptive 1 defines key-frame pairs as $K(d) = \\{(x_i, x_j) \\mid D(x_i, x_j) \\leq d\\}$ with $D(x_i,x_j) = f\\left(\\sum_{s \\in S} \\sum_{k=i+1}^{j} \\|s(x_i)-s(x_k)\\|\\right)$ and $f(z_d)=1/(z_d+\\epsilon)^{\\beta_d}$, so pairs whose cumulative centroid displacement stays within a threshold are treated as key frames. Adaptive 2 replaces displacement with accumulated absolute velocity differences $\\sum_{s \\in S} \\sum_{k=i+1}^{j} |V_s(x_i)-V_s(x_k)|$ with $f(z_v)=1/(z_v+\\epsilon)^{\\beta_v}$. These signals are computed from centroids supplied by a fine-tuned YOLOv8 detector, and the threshold is set indirectly by choosing a target percentage of training frames. The retained frames feed two X3D CNN classifiers, one on RGB and one on optical flow, whose probability outputs are ensembled for the final phase label.","core_discovery":"The central claim is that in robotic anastomosis videos, semantic redundancy tracks kinematic redundancy at the instrument centroids. KAFR computes a distance or velocity-variation signal over tracked tool parts, jaw, wrist, and shaft, and marks frame pairs whose cumulative signal stays below a threshold as similar, keeping only the bounding key frames. Training two X3D CNN streams, one on RGB and one on Farneback optical flow, on those key frames outperforms training on the full 6 fps sequence, with relative accuracy gains of 4.32% on the GJ dataset and 2.05% on the PJ dataset at roughly 10% and 20% of the frames respectively. The authors interpret this as removing frames that inject confusion into a frame-by-frame learner rather than simply compressing the video.","pith_inferences":["A natural extension the paper does not test is online inference: if tool kinematics are computed in real time, KAFR could decide at capture time which frames to store, lowering the storage cost of surgical video archives.","The method's usefulness is bounded by what it tracks: in a phase like Enterotomy, where the untracked tool does the critical cutting, KAFR drops informative frames, so procedures with frequent camera motion or tool occlusion are a risk zone.","Because KAFR selects frames before classification, it could be bolted onto temporal models such as transformers or TCNs rather than only the X3D CNN used here, turning frame selection into a preprocessing step for long-video learning generally.","The reported gains compare against training on all frames at 6 fps; a more direct baseline for the redundancy claim would be an equally reduced uniform sample, which the paper only partially addresses through its MSE comparison."],"forward_implications":["GJ phase segmentation reaches 0.7814 accuracy and 0.7141 F1 with a tenfold frame reduction, so practitioners can train on a small fraction of frames without sacrificing performance.","KAFR makes frame count a tunable resource: at 5% or 1% retention accuracy falls, so the method identifies both a floor for redundant frames and a critical-mass requirement for training.","The two-stream ensemble of RGB and optical flow contributes the largest gains, meaning the selection rule and the multi-stream classifier are complementary components of the reported result.","On the PJ dataset the same procedure transfers with a fivefold reduction, indicating the method is not overfit to one anastomosis type.","Because KAFR removes duplicated and near-static frames, it cuts memory and per-epoch runtime while retaining phase-defining moments, which the paper reports as a practical resource-efficiency benefit."],"supporting_citations":[{"why":"Supplies the PJ dataset, the previous X3D CNN phase-segmentation pipeline, and the 0.8801 baseline accuracy that KAFR must beat.","marker":"[3]"},{"why":"Provides the YOLO detection framework that KAFR uses to find and track surgical tool parts.","marker":"[41]"},{"why":"Defines the X3D architecture used as the classifier on the selected key frames.","marker":"[47]"},{"why":"The Farneback optical-flow algorithm supplies the second stream in the two-stream ensemble.","marker":"[57]"},{"why":"TeCNO results on Cholec80 and Cholec51 serve as the main state-of-the-art comparison baseline.","marker":"[62]"},{"why":"Provides the Cholec80 and Cholec51 datasets and the Endonet plus HHMM comparison accuracy used in the state-of-the-art table.","marker":"[31]"},{"why":"The automated tool-tracking study that underlies the centroid-tracking step of KAFR.","marker":"[46]"}],"fun_headline_variants":["Tool-tracking picks key frames, cutting data by 90% with higher accuracy","Surgical AI needs 10% of frames for better phase detection","KAFR: Track tools, drop redundant frames, improve surgical video AI","Motion-based frame selection boosts surgical video analysis accuracy","One-tenth the frames, higher accuracy: smart frame extraction for surgery"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that consecutive frames in which the tracked tool centroids barely move contain no information needed to classify the surgical phase, so throwing those frames away cannot cost accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Tool-tracking picks key frames, cutting data by 90% with higher accuracy","Surgical AI needs 10% of frames for better phase detection","KAFR: Track tools, drop redundant frames, improve surgical video AI","Motion-based frame selection boosts surgical video analysis accuracy","One-tenth the frames, higher accuracy: smart frame extraction for surgery"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000813,"raw_usage":{"total_tokens":3591,"prompt_tokens":999,"completion_tokens":2592,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":2499}},"tokens_in":615,"tokens_out":2592,"duration_ms":17600,"temperature":1.0,"reasoning_tokens":2499,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:34:45.941772+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same two-stream X3D setup on a surgical video where the phase-defining action is performed by an untracked tool or by camera motion, with the Enterotomy example in Figure 13 as a natural candidate; if KAFR selects fewer frames in those segments than a uniform sampler at equal retention and its accuracy drops to or below the uniform-sampler level, the assumption that low centroid motion equals redundancy is falsified.","supporting_citations":[{"cited_title":"The development of a deep learning model for automated segmentation of the robotic pancreaticojejunostomy","cited_arxiv_id":null,"evidence_quote":"Supplies the PJ dataset, the previous X3D CNN phase-segmentation pipeline, and the 0.8801 baseline accuracy that KAFR must beat."},{"cited_title":"Y ou only look once: Unified, real-time object detection","cited_arxiv_id":null,"evidence_quote":"Provides the YOLO detection framework that KAFR uses to find and track surgical tool parts."},{"cited_title":"X3d: Expanding architectures for efficient video recognition","cited_arxiv_id":null,"evidence_quote":"Defines the X3D architecture used as the classifier on the selected key frames."},{"cited_title":"Two-frame motion estimation based on polynomial expansion","cited_arxiv_id":null,"evidence_quote":"The Farneback optical-flow algorithm supplies the second stream in the two-stream ensemble."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"TeCNO results on Cholec80 and Cholec51 serve as the main state-of-the-art comparison baseline."},{"cited_title":"Machine Learning-Based Automated Assessment of Intracorporeal Suturing in Laparoscopic Fundoplication","cited_arxiv_id":"2412.16195","evidence_quote":"The automated tool-tracking study that underlies the centroid-tracking step of KAFR."}],"review_version":1}