{"id":"26cb7ca1-710a-4c70-9761-df8f398ef7b4","arxiv_id":"2412.04039","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A causal hierarchical attention transformer (adapted from ASFormer) outperforms four prior models on a new RAMIE phase-recognition dataset and on AutoLaparo, although the AutoLaparo baseline comparison is not clearly protocol-matched.","lead":"The paper introduces a new dataset of 27 videos for recognizing surgical phases in robotic-assisted esophagectomy and benchmarks four existing models plus a new causal transformer variant that reportedly improves accuracy and phase-segmentation scores. It matters because automated phase recognition could support surgeons during these complex procedures and enable post-operative analysis, but the comparison on the public AutoLaparo dataset may not be apples-to-apples.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The AutoLaparo 'superiority' column is built on literature baselines rather than reruns under the paper's own Funke-code protocol; without evidence that metric, split, and preprocessing conventions match, the cross-dataset claim is unsubstantiated.","rationale":"The paper's RAMIE experiments are internally consistent: the baselines are implemented and evaluated under the same framework, and the Edit/F1 differences are far larger than the reported across-video standard deviations. The single weakest link in the argument for general superiority is the AutoLaparo comparison, because Table 3 explicitly labels baseline numbers as coming from the original AutoLaparo paper rather than from re-runs under the paper's own protocol. The reader's weakest_assumption identifies exactly this issue, and it is concrete and testable. The paper deserves credit for the dataset contribution and for the controlled RAMIE benchmark, so no rejection is warranted; however, the unqualified abstract claim should be conditioned on either re-running the AutoLaparo baselines or restricting the superiority claim to the RAMIE dataset.","tokens_in":7138,"tokens_out":4900,"duration_ms":52178,"concrete_test":"Run all four baselines on AutoLaparo with the same 10/4/9 split, the same ResNet feature-extraction pipeline, and the same Funke et al. metric code used for the proposed model, then recompute Table 3. If baseline Accuracy, Edit, or F1 values change by more than a few points, or if any baseline meets or exceeds the reported 83.18 Accuracy, the cross-dataset superiority claim must be withdrawn or qualified to RAMIE-only. As a lighter check, verify from the code whether the original [12] metric implementations coincide with Funke's implementations for the metrics reported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that the proposed causal hierarchical attention model surpasses existing models. On the new RAMIE data, the comparison is plausibly controlled: Section 3.2 says the four baselines were run from original author code, and Section 3.4 says all models were evaluated with the Funke et al. codebase, so the very large Edit/F1 gaps in Tables 1 and 2 are internally meaningful. The load-bearing weak point is the AutoLaparo evidence in Table 3. The caption states 'baseline results from [12]', i.e., the baseline rows were not recomputed under the same protocol as the proposed model. Funke et al. [21] is cited precisely because metric implementations vary across surgical-phase papers (per-video versus global accuracy, phase-averaged Jaccard, segment matching for Edit/F1), and the original AutoLaparo paper predates the Funke codebase. Differing splits, preprocessing, feature extractors, or metric definitions can shift scores by several points; here only means are available for baselines, while the proposed model's own standard deviations are large (Accuracy 83.18 ± 9.75). Without re-running SV-RCNet, TMRNet, TeCNO, and Trans-SVNet under the paper's evaluation protocol, the claimed AutoLaparo superiority is not established. The RAMIE result may still stand, but the abstract's unqualified 'superior performance compared to existing models' leans on this uncontrolled comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a new surgical phase recognition dataset for the thoracic phase of robotic-assisted minimally invasive esophagectomy (RAMIE), comprising 27 videos with 13 annotated phases and a 14/4/9 train/validation/test split. Using this dataset, the authors benchmark four existing surgical phase recognition models (SV-RCNet, TMRNet, TeCNO, Trans-SVNet) and propose a new temporal model: an encoder-decoder transformer with causal hierarchical attention, adapted from ASFormer, trained with a cross-entropy plus smoothing loss. On RAMIE, the proposed model achieves large improvements in Edit score and F1@25/50/75 over the baselines, but its accuracy is comparable to or slightly lower than TeCNO. The paper also reports results on the public AutoLaparo dataset, where the proposed model's metrics exceed numbers taken from the original AutoLaparo paper. The authors discuss remaining challenges such as over-segmentation near phase transitions, class imbalance, and limitations of current evaluation metrics.","tokens_in":7447,"tokens_out":6152,"duration_ms":59131,"significance":"If the empirical results hold, the paper makes a useful contribution by providing a new dataset for a surgical procedure with complex, non-sequential phase transitions, and by showing that a causal transformer variant can substantially improve temporal segmentation metrics on that dataset. Strengths include the use of the Funke et al. evaluation codebase for all RAMIE experiments, the comparison of several baseline models run from original author code, the inclusion of Edit score and segment-level F1 metrics in addition to frame-level accuracy, and the qualitative analysis of error patterns. However, the significance is currently limited by the lack of ablations supporting the attribution of gains to the causal mechanism, by an uncontrolled AutoLaparo baseline comparison that relies on literature numbers, and by the small, single-annotator RAMIE test set. These issues are fixable and do not undermine the dataset contribution, but they must be addressed before the central claim of superiority can be accepted.","major_comments":[{"comment":"The AutoLaparo superiority claim is not supported by controlled evidence. Table 3 states that baseline results are taken from [12] rather than re-run under the evaluation protocol used for the proposed model (Funke et al. codebase, §3.4). Because metric definitions, train/validation splits, preprocessing, and feature extractors differ across surgical-phase papers, the baseline numbers in [12] are not directly comparable to the proposed model's scores. Moreover, §3.1.2 says the AutoLaparo split is 10/4/9 videos, which sums to 23 rather than the stated 21, and gives the same frame count (12,056) for validation and test; this internal inconsistency must be resolved. Re-running SV-RCNet, TMRNet, TeCNO, and Trans-SVNet under the paper's protocol, or at minimum reporting the exact split and metric implementations with baseline numbers, is necessary before claiming superiority on this dataset.","section":"§3.5, Table 3"},{"comment":"The Discussion attributes the observed gains to the causal hierarchical attention mechanism and the smoothing loss, but no ablation isolates these components. The paper does not compare the proposed causal encoder-decoder against its non-causal ASFormer counterpart, nor against variants with λ=0 in Eq. (1), nor against configurations with different numbers of decoder layers. Without such ablations, the central claim that the causal design is responsible for the large Edit/F1 improvements on RAMIE is an unsupported attribution. Adding at least a causal-vs-non-causal comparison and a smoothing-loss on/off comparison would strengthen the paper's main conclusion.","section":"§4; §3.3.2"},{"comment":"The RAMIE benchmark has limited statistical grounding. The ground truth was produced by a single annotator (a PhD student guided by a medical PhD student and surgeon), with no inter-annotator agreement reported, which is important for a dataset intended as a community benchmark. The test set contains only 9 videos, and several metrics in Table 1 overlap within one standard deviation (e.g., Accuracy: Ours 78.28 ± 4.42 vs. TeCNO 78.46 ± 3.97; Jaccard: Ours 61.94 ± 7.24 vs. TeCNO 58.34 ± 4.75). The paper reports no per-video significance tests. The large Edit/F1 margins in Table 2 are encouraging, but the broad claim of superiority should be restricted to metrics where the gap is clear, or supported by significance testing.","section":"§3.1.1; Tables 1 and 2"},{"comment":"The abstract's unqualified statement that the proposed model 'demonstrates superior performance compared to existing models' is not consistent with all reported metrics. On the RAMIE dataset, the proposed model's Accuracy (78.28) is numerically lower than TeCNO's (78.46) in Table 1. The claim should be qualified to temporal segmentation metrics (Edit score, F1@τ), or the accuracy result should be discussed explicitly; otherwise the headline claim overreaches the evidence.","section":"Abstract; §3.4"}],"minor_comments":[{"comment":"The loss notation is confusing: Eq. (1) uses H(S(p_{y_t,t}), y_t), but cross-entropy is written as if between a probability vector and a label without a one-hot encoding, and Eq. (2) defines Δ_t per class c while Eq. (1) applies clamp(Δ_t², 0, 16) without a class subscript. Please clarify the dimensions and define the softmax subscript.","section":"§3.3.2, Eq. (1)-(2)"},{"comment":"The text says baselines were implemented from original author code 'maintaining all original settings', but §3.4 says all models were evaluated with the Funke et al. codebase. Please clarify how the original settings interact with the Funke evaluation harness, especially regarding input preprocessing and feature extraction.","section":"§3.2, §3.4"},{"comment":"The baseline rows in Table 3 report only means, while the proposed model reports mean ± standard deviation. If the baseline numbers must remain from [12], at least state this explicitly in the text and note that standard deviations are unavailable for the baselines.","section":"Table 3"},{"comment":"There are minor consistency issues in notation and naming: 'ASformer' and 'ASFormer' are used interchangeably, 'Trans-SVnet' appears in Tables 1-2 while 'Trans-SVNet' is used elsewhere, and Figure 3's l, T_l, h_l notation is defined only in the caption. A consistent notation pass would improve readability.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper's empirical core on RAMIE is plausible and the dataset is a useful contribution. The main risks are the uncontrolled AutoLaparo baseline comparison and the lack of ablations; both are fixable within the scope of a revision. I would also ask the authors to correct the AutoLaparo split arithmetic and to state whether the dataset will be released, since the benchmark contribution depends on it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real contribution here is the RAMIE dataset: 27 videos, 13 thoracic phases, with a sensible split and annotations guided by an expert surgeon. That alone justifies attention. The authors also did the right thing by running the four baselines from original code and evaluating everything with the Funke et al. metrics codebase, so the RAMIE head-to-head is internally controlled. The proposed model, a causal-masked variant of ASFormer, is a modest but transparent modification, and the gains on the RAMIE test set are large and consistent for edit score and F1: 59.50 vs. the best baseline's 15.63 on edit, 58.42 vs. 19.34 on F1@25, with non-overlapping standard deviations. I believe those numbers reflect a real improvement on this dataset, not just metric noise. Accuracy is essentially tied with TeCNO, which is fine—the authors are targeting temporal coherence.\n\nThe soft spots are real but concentrated. The AutoLaparo comparison in Table 3 is the load-bearing weakness: baseline numbers are taken from the original paper rather than re-run under the same Funke-code protocol, and the stress-test note is right that metric definitions, splits, and preprocessing can shift scores by several points. The authors' own standard deviation on accuracy is ±9.75, so the 4.9-point gap over Trans-SVNet is not clearly significant. Also missing: ablation of the causal mask, statistical significance tests, and a discussion of how the smooth-loss weight was chosen. The RAMIE ground truth comes from a single annotator with nine test videos, which is a limitation but not fatal for a pilot benchmark. The dataset and code are not yet public, which makes independent verification harder.\n\nDespite the AutoLaparo issue, the paper is honest about its scope, the discussion acknowledges over-segmentation and phase-transition errors, and the methodology is described in enough detail to reproduce once the data ships. This deserves a serious referee: a good reviewer can push for the AutoLaparo rerun, ablations, and significance testing. For my own work, I would cite the RAMIE dataset once it is available; the model's novelty is not the selling point. I'd bring it to a reading group to discuss metric harmonization across surgical-phase papers, which is exactly the kind of practical issue the field needs to face.","headline":"A useful new RAMIE phase-recognition dataset and a plausible causal-transformer adaptation, but the AutoLaparo comparison rests on literature baselines, so the unqualified superiority claim is not yet earned.","tokens_in":7971,"tokens_out":1652,"would_cite":true,"duration_ms":18059,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A causal transformer with hierarchical attention outperforms four prior models on a new surgical phase recognition benchmark for robotic esophagectomy.","keywords":["surgical phase recognition","robotic-assisted esophagectomy","RAMIE","causal attention","hierarchical transformer","temporal action segmentation","AutoLaparo","surgical video analysis"],"falsifier":"Re-run SV-RCNet, TeCNO, TMRNet, and Trans-SVNet with the same training/validation/test splits and the same evaluation code on both RAMIE and AutoLaparo; if the reported Edit Score gaps shrink or reverse, the claimed superiority is an artifact of protocol mismatch.","tokens_in":6968,"feed_emoji":"🤖","tokens_out":6312,"duration_ms":56149,"temperature":0.7,"pith_summary":"The paper sets out to show that surgical phase recognition for robotic-assisted minimally invasive esophagectomy (RAMIE) is not served well by models built for more sequential procedures, and that a causal transformer with hierarchical attention closes the gap. To test this, the authors assembled a 27-video RAMIE dataset of the thoracic phase, with 13 annotated phases, and benchmarked four established methods plus their own model. Their model reports the best results on most metrics, with particularly large gains on segment-level metrics: Edit Score 59.50 and F1@25 58.42, versus 13.15 and 17.79 for the strongest baseline TeCNO. The stated aim is to provide a foundation for intraoperative support and postoperative analysis in RAMIE.","feed_headline":"Causal transformer tops surgical phase recognition in new RAMIE data","feed_subtitle":"On the non-sequential phases of RAMIE, edit score jumps from 13.15 to 59.50 versus the best prior model.","key_machinery":"The central object is a causal transformer with one encoder and three decoders, each with ten blocks built from causal dilated convolutions and masked self-attention. At layer $l$, the query tensor $Q_l \\in \\mathbb{R}^{T_l \\times d \\times h_l}$ and the key tensor $K_l \\in \\mathbb{R}^{T_l \\times d \\times 2h_l}$ operate on a halved sequence length with doubled head dimension, and a causal mask forces each position to attend only to earlier frames. The mechanism is adapted from ASformer, an action-segmentation transformer, and it is coupled with a loss that combines per-frame cross-entropy with a clamped temporal smoothness penalty to reduce fragmented phase predictions.","core_discovery":"The central claim is that adding a causal mask to hierarchical self-attention in an encoder-decoder transformer substantially improves phase recognition on procedures with repetitive phases and non-sequential transitions. On the new RAMIE dataset, the model reaches 78.28 accuracy and an Edit Score of 59.50, far above the compared baselines; on AutoLaparo it reports 83.18 accuracy versus 78.29 for Trans-SVNet. The paper attributes the gains to causal modeling of temporal order, multi-layer decoder refinement, and a smoothing loss that suppresses over-segmentation.","pith_inferences":["Editorial inference: If the AutoLaparo results hold under a fully re-run protocol, the same causal design may transfer to other multi-phase laparoscopic procedures with non-canonical phase orders, such as complex hysterectomy variants.","Editorial inference: The reported sensitivity to phase boundaries suggests an extension that predicts transition points explicitly, or that weights the loss by phase risk, could push clinically useful accuracy further.","Editorial inference: Because the RAMIE labels come from a single annotator, a natural next step is a multi-annotator study to measure how much of the performance gap is annotation noise rather than model behavior."],"forward_implications":["If the causal architecture is the reason for the gains, temporal causality should become a standard design choice for phase-recognition models on non-sequential surgical workflows.","The RAMIE dataset and its 13-phase annotation scheme give the community a benchmark for procedures with phase-order variability, where existing benchmarks are largely sequential.","Further improvements should target phase boundaries and short phases such as right pleural and aortopulmonary lymph node dissection, where most classification errors concentrate.","Segment-level metrics like Edit Score and F1@tau should be reported alongside frame accuracy, because frame accuracy alone did not reflect the model's large improvement."],"supporting_citations":[{"why":"Supplies the AutoLaparo dataset and the published baseline numbers used in the cross-dataset comparison.","marker":"[12]"},{"why":"TeCNO is the strongest baseline on RAMIE and its multi-stage temporal convolution architecture is the main comparison point.","marker":"[7]"},{"why":"SV-RCNet is one of the four benchmarked baselines, combining ResNet features with LSTM temporal modeling.","marker":"[13]"},{"why":"TMRNet is one of the benchmarked baselines and represents multi-scale temporal memory approaches.","marker":"[14]"},{"why":"Trans-SVNet is the transformer-based baseline whose AutoLaparo accuracy is compared against the proposed model.","marker":"[15]"},{"why":"ASformer is the action-segmentation transformer whose encoder-decoder and hierarchical attention structure the proposed model adapts.","marker":"[17]"},{"why":"Source of the smoothing loss used to penalize over-segmentation and improve segment-level metrics.","marker":"[20]"},{"why":"Evaluation codebase used to ensure consistent metric implementation across all models in the study.","marker":"[21]"},{"why":"Defines the edit score metric used to measure segment-level sequence similarity.","marker":"[22]"},{"why":"Defines the F1@tau metric used to compare segment overlap against ground truth at different thresholds.","marker":"[23]"}],"fun_headline_variants":["Causal transformer boosts RAMIE phase Edit Score from 13 to 59","New dataset and causal model beat existing RAMIE phase recognizers","Causal hierarchy transformer handles RAMIE's non-sequential phases","RAMIE phase recognition jumps to 78% accuracy with causal attention"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The AutoLaparo comparison assumes the baseline numbers reproduced from the original paper were computed under the same metric implementation and data split, since those baselines were not re-run with the authors' evaluation code.","fun_headline_variants_meta":{"raw":{"variants":["Causal transformer boosts RAMIE phase Edit Score from 13 to 59","New dataset and causal model beat existing RAMIE phase recognizers","Causal hierarchy transformer handles RAMIE's non-sequential phases","RAMIE phase recognition jumps to 78% accuracy with causal attention"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000831,"raw_usage":{"total_tokens":3552,"prompt_tokens":793,"completion_tokens":2759,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":409,"completion_tokens_details":{"reasoning_tokens":2684}},"tokens_in":409,"tokens_out":2759,"duration_ms":18765,"temperature":1.0,"reasoning_tokens":2684,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:48:40.634848+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run SV-RCNet, TeCNO, TMRNet, and Trans-SVNet with the same training/validation/test splits and the same evaluation code on both RAMIE and AutoLaparo; if the reported Edit Score gaps shrink or reverse, the claimed superiority is an artifact of protocol mismatch.","supporting_citations":[{"cited_title":"Autolaparo: A new dataset of integrated multi-tasks for image-guided surgical automation in laparoscopic hysterectomy","cited_arxiv_id":null,"evidence_quote":"Supplies the AutoLaparo dataset and the published baseline numbers used in the cross-dataset comparison."},{"cited_title":"Tecno: Surgical phase recognition with multi-stage temporal convolutional net- works","cited_arxiv_id":null,"evidence_quote":"TeCNO is the strongest baseline on RAMIE and its multi-stage temporal convolution architecture is the main comparison point."},{"cited_title":"Sv-rcnet: workflow recognition from surgical videos using recurrent convolutional network","cited_arxiv_id":null,"evidence_quote":"SV-RCNet is one of the four benchmarked baselines, combining ResNet features with LSTM temporal modeling."},{"cited_title":"Temporal memory relation network for workflow recognition from surgical video","cited_arxiv_id":null,"evidence_quote":"TMRNet is one of the benchmarked baselines and represents multi-scale temporal memory approaches."},{"cited_title":"Trans-svnet: Accurate phase recognition from surgical videos via hybrid embedding aggregation transformer","cited_arxiv_id":null,"evidence_quote":"Trans-SVNet is the transformer-based baseline whose AutoLaparo accuracy is compared against the proposed model."},{"cited_title":"Ms-tcn: Multi-stage temporal convolutional network for action seg- mentation","cited_arxiv_id":null,"evidence_quote":"Source of the smoothing loss used to penalize over-segmentation and improve segment-level metrics."},{"cited_title":"Segmental spatiotemporal cnns for fine- grained action segmentation","cited_arxiv_id":null,"evidence_quote":"Defines the edit score metric used to measure segment-level sequence similarity."}],"review_version":1}