{"id":"3ca247e3-ecf0-43e9-aeb2-272682457e78","arxiv_id":"2502.07431","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A transformer-based model with a temporal progress head reaches 72.91% accuracy on the new ACL27 arthroscopic phase dataset and 92.4% on Cholec80, establishing an early benchmark for arthroscopy.","lead":"This paper introduces ACL27, a dataset of 27 annotated ACL reconstruction videos, and a transformer-based model for surgical phase recognition that also predicts a Surgical Progress Index. It reports 72.91% phase-recognition accuracy on ACL27 and 92.4% on the public Cholec80 benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ACL27 benchmark rests on unvalidated single-pass labels; without inter-rater reliability the 72.91% accuracy and its margin over baselines cannot be separated from label noise.","rationale":"The reader's weakest_assumption—that ACL27 label quality and missing-phase SPI imputation are load-bearing—is exactly where I land. All benchmark numbers, the claimed first benchmark for arthroscopy, and the model-vs-baseline ranking in Table 3 depend on those labels. No amount of architectural novelty fixes an unvalidated reference: if two experts disagree on a substantial fraction of the 1 Hz labels, then the reported accuracy is an estimate of agreement with one annotator, and the six-point advantage over Trans_SVNet could be an artifact of temporal-smoothing or SPI-based priors that happen to align with that annotator's noise. I considered whether the omitted Opera results or the inconsistent ablation narrative (Table 4 rows 2 and 3 are swapped in the text) are more decisive, but those affect attribution of gains, not the existence of the benchmark. I also considered the SPI's conceptual weakness as normalized elapsed time; however, the classification claim is independent of the usefulness of SPI, so the label-reliability issue is the single point on which the strongest claim rises or falls. The proposed test—second-expert reannotation on a subset with consensus-label evaluation—would settle whether the concern lands. If agreement is high and the margin survives on consensus labels, the central claim is supported; if not, the benchmark needs to be re-established. Since the reader already set CONDITIONAL and this concern reinforces that condition, I do not change the verdict.","tokens_in":11657,"tokens_out":6550,"duration_ms":61696,"concrete_test":"Ask the authors to release ACL27 labels and have a second arthroscopic surgeon independently annotate at least 5 videos (covering all five phases) at the same 1 Hz frame rate. Compute frame-level Cohen's kappa (or Gwet's AC1) between the original and new annotations, then recompute accuracy for the proposed model and the three strongest baselines on the subset of frames where both annotators agree. If the proposed model's margin over Trans_SVNet collapses or falls within the disagreement rate, the benchmark claim is not supported. As a secondary check, rerun the full pipeline on the 12 complete videos without SPI imputation and compare the resulting accuracy and SPI error to the reported 72.91% and 10.6%.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim—72.91% accuracy and a roughly six-point lead over Trans_SVNet on ACL27 (Table 3)—is computed entirely against the labels described in Section 3.1. Those labels were created 'in close collaboration with clinical experts' at one label per second, but the paper reports no annotator count, no inter-rater reliability, no adjudication rule, and no quality-control measure on the ~13-hour corpus. The SPI targets in Section 3.3 are additionally imputed for videos with missing phases using average transition points from the 12 complete videos, so the auxiliary regression signal is partly derived from the same phase statistics rather than from an independent progress measure. If the frame-level labels are noisy—plausible given the paper's own emphasis on blur, occlusion, and fluid artifacts (Fig. 1)—the reported accuracy measures agreement with a noisy reference, and the ranking against Trans_SVNet, MTRCNet, and TeCNO on the same noisy labels may not persist under cleaner annotation. The reported 72.91 ± 11.76% (Section 6) also indicates high fold-to-fold variance over only 27 videos, making the headline number fragile. Since the dataset and code are not yet public, an independent check of the labels is currently impossible.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ACL27, a new dataset of 27 arthroscopic anterior cruciate ligament reconstruction videos with five surgical phases labeled at one label per second, and proposes a transformer-based phase recognition model with temporal-aware frame-wise features and a continuous Surgical Progress Index (SPI). The authors report 72.91% accuracy, 72.86% precision, and 57.39% Jaccard Index on ACL27, and 92.4% accuracy on the Cholec80 benchmark. They also report SPI errors of 10.6% and 9.86% on the two datasets. The paper positions the model as the first benchmark for arthroscopic phase recognition and claims generalizability via the Cholec80 experiments.","tokens_in":11900,"tokens_out":5200,"duration_ms":49594,"significance":"If the ACL27 labels are reliable, the dataset fills a genuine gap: arthroscopic phase recognition is essentially unexplored compared with laparoscopy, and a public dataset with baselines would be a useful community resource. The Cholec80 comparison is a sensible external validity check and the reported accuracy is competitive with published methods. The central accuracy number is not circular, since the phase labels are expert annotations independent of the model. The SPI is a lightweight auxiliary supervision idea that is easy to adopt. However, the paper's load-bearing claims currently rest on unvalidated single-pass labels, an SPI target that is partly constructed from average phase statistics, and reimplemented baselines without release details. The dataset and code are promised but not yet available, so independent verification is not possible at this stage.","major_comments":[{"comment":"The ACL27 benchmark is computed entirely against labels that are described only as created 'in close collaboration with clinical experts' at one label per second. The paper reports no number of annotators, no inter-rater reliability statistic, no adjudication rule, and no quality-control measure. Given the paper's own emphasis on blur, occlusion, and fluid artifacts, label noise is a real risk; the reported 72.91% accuracy and the roughly six-point margin over Trans_SVNet could partly reflect agreement with a noisy reference. Please add an inter-rater reliability study on a subset (e.g., Cohen's kappa or Krippendorff's alpha) and a description of how ambiguous frames were resolved.","section":"Section 3.1, Table 3"},{"comment":"The SPI target for videos with missing phases is constructed using average transition points computed from the 12 complete ACL27 videos, and the reported SPI error of 10.6% is measured against this adjusted target. The SPI error is therefore not an independent validation of the model's ability to estimate true surgical progress: for missing-phase videos, the target already incorporates dataset-level phase statistics. Please report SPI error separately on complete videos only, and/or validate the imputation by hiding phases in complete videos and comparing the adjusted SPI against the true SPI. The error metric (apparently MAE) should also be stated explicitly.","section":"Section 3.3, Eq. (2), Section 4.2"},{"comment":"The evaluation protocol for the 5-round cross-validation is ambiguous. The text says the datasets were randomly divided into five equal partitions, but for ACL27 it also says 18 videos were used for training and 9 for validation; 27 videos cannot be divided into five equal partitions of this size. This ambiguity affects the interpretation of the ablation results in Table 4, including the headline robust configuration accuracy of 76.71 ± 2.44. Please specify exactly how the folds were generated, whether the 18/9 split refers only to the non-cross-validation experiments, which random seeds were used, and how the reported mean and standard deviation were computed.","section":"Section 3.4, Section 4.3"},{"comment":"The comparative baselines (Trans_SVNet, MTRCNet, TeCNO, Opera) are author reimplementations, but the paper provides no hyperparameters, training schedules, or code for these baselines, and no indication of whether published checkpoints or official implementations were used. The central claim that the proposed model outperforms existing methods on ACL27 depends on these reimplementations being faithful. Please provide the full experimental configuration for each baseline and, ideally, release the code used for the comparisons. There is also an internal inconsistency: the text says MTRCNet achieved 58.75%, while Table 3 lists 60.68% for the Seq=10s row and Section 6 cites 60.68%.","section":"Section 4.2, Table 3"},{"comment":"The paper reports '72.91 ± 11.76%' for ACL27 accuracy, but Table 3 lists 72.91% as a non-cross-validation result with no standard deviation. It is unclear whether the ±11.76 is a per-video standard deviation, a standard deviation across non-CV runs, or something else. This is a load-bearing number in the abstract and conclusion, and the large spread deserves explicit explanation. Please state which split the 72.91% figure comes from and define the uncertainty interval in all places where it appears.","section":"Sections 5 and 6"}],"minor_comments":[{"comment":"The text misassigns the ablation rows: it says 'when spatio-temporal features are added (second row)', but the second row is the SPI-only configuration, and the third row is the spatio-temporal-features-only configuration. Please correct the row references.","section":"Section 4.3, Table 4"},{"comment":"The phase label 'ACL Reko' in the figure appears to be a truncated version of 'ACL Reconstruction'; please use the full phase name for clarity.","section":"Figure 1"},{"comment":"Both Section 5 and Section 6 are titled 'Discussion', which is confusing. Consider merging them or giving distinct titles.","section":"Sections 5 and 6"},{"comment":"There is a typo in 'benchmark the proposed method with tate-of-the-art approaches'; it should read 'state-of-the-art'.","section":"Section 1, Contributions"},{"comment":"The notation is inconsistent: the SPI is initially defined for surgery k, but the adjusted formula uses k in the first term and then switches to 'video n' in the surrounding text. Please use a single video index throughout.","section":"Section 3.3, Eqs. (1) and (2)"},{"comment":"The phrase 'we use only videos that contain all predefined surgical phases for baseline computation (12 videos in ACL27 and 65 in Cholec80)' should be clarified: are these the videos used to compute the average SPI transitions in Table 2, and are the remaining videos used only with the adjusted SPI target? The current wording is easy to misread as applying to the whole experimental benchmark.","section":"Section 3.3"},{"comment":"The architecture description says a transformer layer with 2 heads is used for temporal-aware feature refinement, but the number of layers, hidden dimensions, and the exact roles of the SFE and TCE are not specified. Please add a short architectural table or precise configuration so the method is reproducible.","section":"Section 3.2, Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable fit for a surgical data science / computer-assisted surgery venue. The main risk is not the novelty of the method but the unverified dataset labels and the constructed SPI targets; these are fixable with additional experiments. I would encourage the editor to treat dataset and code release as a condition of acceptance, since the ACL27 benchmark cannot be assessed otherwise. There are no obvious citation or attribution problems."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the one thing to know is that the contribution is the ACL27 dataset, not the model. The model is a solid but incremental transformer variant; the SPI is normalized elapsed time. The dataset deserves a home in the literature, but the paper needs a round of major revision before I'd trust the benchmark numbers.\n\nWhat's genuinely new and good: ACL27, 27 labeled ACL reconstruction videos with five phases, is the first public arthroscopic phase-recognition benchmark. That is a real gap, and a useful resource if the labels hold up. The model is not novel in any architectural sense—ResNet-50 plus transformers plus a regression head—but the ablation suggests both the spatio-temporal features and the SPI auxiliary task help. The Cholec80 number (92.4% accuracy, comparable to SKiT and LoViT) is important evidence that the method is competitive on an established benchmark, which raises my confidence in the architecture.\n\nSoft spots, in proportion: The labels are a single expert pass at 1 Hz, with no inter-rater reliability, no annotator count, no adjudication, and no quality control. That is the load-bearing weakness. The SPI is t/T_k, normalized time, with missing-phase adjustment using average transition points from complete videos; the reported 10.6% error is partly self-referential. The baseline comparisons are author reimplementations without tuning details or release, so the six-point gap over Trans-SVNet on ACL27 is unverifiable. Section 4.3's text misassigns rows in Table 4—it says row 2 adds spatio-temporal features when the table says it adds only SPI. Also, the abstract promises 'publicly available dataset and code' while Section 3.1 says 'upon acceptance'; the paper should be consistent about release timelines. The 72.91 ± 11.76% headline in Section 6 sits oddly against the 76.71 ± 2.44 cross-validated ablation, and the manuscript should explain which is which.\n\nWho is this for: the surgical video analysis community, particularly orthopedics. A reader who wants to benchmark arthroscopic phase recognition will need this dataset. I'd bring it to a reading group to critique the evaluation methodology, not to copy the model.\n\nRecommendation: send to peer review, with a condition that a serious referee asks for artifact release and label reliability. The dataset alone justifies referee time.","headline":"ACL27 is the real contribution—a first public arthroscopic phase-recognition dataset—and the model is a competent but incremental transformer variant; the SPI is normalized time, and the central accuracy claim needs label-reliability evidence before it anchors a benchmark.","tokens_in":12449,"tokens_out":3925,"would_cite":true,"duration_ms":36769,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new 27-video ACL surgery dataset and a transformer model with a continuous progress index establish the first benchmark for phase recognition in arthroscopy.","keywords":["surgical phase recognition","arthroscopy","ACL reconstruction","transformer","Surgical Progress Index","spatio-temporal features","ACL27 dataset","surgical video analysis"],"falsifier":"Have several independent surgeons label a subset of ACL27 frames and measure agreement on phase boundaries; if agreement is poor, or if a model retrained on SPI targets computed without the missing-phase imputation loses most of its advantage, the central benchmark claim is overturned. A simpler check: restrict evaluation to the twelve videos containing all five phases and see whether 72.91% accuracy holds.","tokens_in":11460,"feed_emoji":"🩺","tokens_out":4674,"duration_ms":40193,"temperature":0.7,"pith_summary":"The paper aims to show that surgical phase recognition works for arthroscopy, a domain with blurry frames, fluid occlusion, and narrow fields of view. To do this it introduces ACL27, a dataset of 27 ACL reconstruction videos labeled into five phases at one label per second, and trains a transformer-based model that combines ResNet-50 spatial features with temporal context and an auxiliary Surgical Progress Index (SPI). The model reaches 72.91% accuracy on ACL27, ahead of adapted laparoscopic methods, and 92.4% accuracy on Cholec80, matching recent state-of-the-art. The SPI predicts surgery progression with 10.6% error on ACL27 and 9.86% on Cholec80. If these results hold, arthroscopy becomes a measurable, trainable domain for automated surgical workflow analysis.","feed_headline":"First arthroscopic phase-recognition benchmark reports 72.9% accuracy","feed_subtitle":"A transformer plus a surgical progress index lets computers track ACL surgery phases and estimate remaining time.","key_machinery":"The load-bearing mechanism is temporal-aware frame-wise feature extraction: a ResNet-50 turns each 240x240 frame into a 2048-dimensional vector, a two-head transformer layer refines these vectors using neighboring frames, and a Spatial Feature Encoder plus Temporal Context Encoder turn the sequence into spatio-temporal features for an 80-second transformer branch. Parallel to phase classification, the Surgical Progress Index (SPI) regresses t/T_k, the fraction of surgery elapsed, with missing phases repaired by average transition points from complete videos. The SPI gives the model global temporal context without processing the whole video, which is what lets a discrete phase classifier suppress noisy frame-level errors.","core_discovery":"The paper's central claim is that spatio-temporal features plus a continuous progress signal make arthroscopic phase recognition tractable despite the domain's visual noise. The evidence is the ACL27 benchmark and the ablations: with neither spatio-temporal features nor SPI the model scores 66.38% accuracy; adding either helps, and adding both reaches 76.71% under five-fold cross-validation. The authors also claim that the model is the first benchmark for arthroscopic phase recognition and that its SPI output, computed as the elapsed fraction t/T_k of the surgery and adjusted for missing phases, gives a clinically useful estimate of remaining procedure time.","pith_inferences":["If a second expert labeled the same ACL27 videos, low inter-rater agreement on phase boundaries would put an upper bound on what any model can learn from these labels, so the 72.91% number should be read with that caveat.","Because SPI is derived from time, the model may learn to lean on elapsed time rather than visual content; a test that shuffles or jitters frame order would reveal how much of the accuracy is time-based.","The missing-phase imputation using average transition points could inflate SPI accuracy on videos that start or end mid-procedure; evaluating only the twelve complete videos would give a stricter estimate.","The same dataset could support boundary-level evaluation (time to phase transitions) and semi-automated labeling tools, both of which the paper names as future work but does not test."],"forward_implications":["ACL27 gives the arthroscopy community a fixed 27-video benchmark with per-second phase labels, so future methods can be compared on the same data.","SPI predictions with roughly 10% error make automated remaining-duration estimation possible, which could help operating room scheduling and patient flow.","The ablation results imply that future arthroscopic phase models should include both temporal feature fusion and a continuous progress head.","The Cholec80 result (92.4% accuracy) suggests the same architecture transfers to laparoscopy without retraining for a new instrument set."],"supporting_citations":[{"why":"EndoNet establishes CNN-plus-HMM phase recognition on Cholec80 and is one of the external benchmarks the model must beat.","marker":"[3]"},{"why":"MTRCNet provides a multi-task recurrent baseline with correlation loss, evaluated here on both ACL27 and Cholec80.","marker":"[14]"},{"why":"TeCNO supplies a temporal-convolution baseline for ACL27 and a Cholec80 comparison point.","marker":"[15]"},{"why":"OperA is the attention-regularized transformer baseline used for Cholec80 comparison.","marker":"[17]"},{"why":"Trans-SVNet is the main ACL27 comparison model, evaluated at sequence lengths of 30, 60, and 80 seconds.","marker":"[18]"},{"why":"LoViT motivates temporal-aware feature extraction and serves as a Cholec80 state-of-the-art comparison.","marker":"[19]"},{"why":"SKiT provides the strongest Cholec80 accuracy (92.5%) against which the model's 92.4% is compared.","marker":"[20]"}],"fun_headline_variants":["First arthroscopic phase benchmark hits 72.9% accuracy","ACL27: new dataset enables 72.9% phase recognition","SPI estimates surgery progress within 10.6% error","Transformer model tracks ACL phases with 72.9% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ACL27 ground truth is one expert's per-second labeling with no second rater, and for videos with missing phases the SPI targets are filled in using average transition points from the twelve complete videos, so the benchmark numbers depend on those labels and that imputation being accurate.","fun_headline_variants_meta":{"raw":{"variants":["First arthroscopic phase benchmark hits 72.9% accuracy","ACL27: new dataset enables 72.9% phase recognition","SPI estimates surgery progress within 10.6% error","Transformer model tracks ACL phases with 72.9% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000803,"raw_usage":{"total_tokens":3544,"prompt_tokens":976,"completion_tokens":2568,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":2495}},"tokens_in":592,"tokens_out":2568,"duration_ms":18402,"temperature":1.0,"reasoning_tokens":2495,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T12:46:56.751714+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have several independent surgeons label a subset of ACL27 frames and measure agreement on phase boundaries; if agreement is poor, or if a model retrained on SPI targets computed without the missing-phase imputation loses most of its advantage, the central benchmark claim is overturned. A simpler check: restrict evaluation to the twelve videos containing all five phases and see whether 72.91% accuracy holds.","supporting_citations":[{"cited_title":"Multi-task recurrent convolutional network with correlation loss for surgical video analysis","cited_arxiv_id":null,"evidence_quote":"MTRCNet provides a multi-task recurrent baseline with correlation loss, evaluated here on both ACL27 and Cholec80."},{"cited_title":"Tecno: Surgical phase recognition with multi-stage temporal convolutional networks","cited_arxiv_id":null,"evidence_quote":"TeCNO supplies a temporal-convolution baseline for ACL27 and a Cholec80 comparison point."},{"cited_title":"Opera: Attention-regularized transformers for surgical phase recognition","cited_arxiv_id":null,"evidence_quote":"OperA is the attention-regularized transformer baseline used for Cholec80 comparison."},{"cited_title":"Trans-svnet: Accurate phase recognition from surgical videos via hybrid embedding aggregation transformer","cited_arxiv_id":null,"evidence_quote":"Trans-SVNet is the main ACL27 comparison model, evaluated at sequence lengths of 30, 60, and 80 seconds."},{"cited_title":"LoViT: Long Video Transformer for Surgical Phase Recognition","cited_arxiv_id":"2305.08989","evidence_quote":"LoViT motivates temporal-aware feature extraction and serves as a Cholec80 state-of-the-art comparison."},{"cited_title":"Skit: a fast key information video transformer for online surgical phase recognition","cited_arxiv_id":null,"evidence_quote":"SKiT provides the strongest Cholec80 accuracy (92.5%) against which the model's 92.4% is compared."}],"review_version":1}