{"id":"cad8f89b-57d2-4ee4-89eb-b78e15bf5287","arxiv_id":"2506.21330","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A hierarchical input-dependent state space model with local and global blocks sets new state-of-the-art phase recognition results on Cholec80, MICCAI2016, and Heichole.","lead":"This paper builds a hierarchical state space model for recognizing surgical phases in long videos, reporting large accuracy gains over prior methods on three public benchmarks. The work matters because it points to a way to process entire surgical videos in one pass instead of cutting them into windows, which could improve automated surgical workflow analysis.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's central full-video claim is undermined by §V-B: the temporal input is described as downsampled to 100 frames per video, which is not a full-length surgical video; this must be resolved before the SOTA numbers can support the holistic long-video claim.","rationale":"The reader identified the PPN-based windowing as the weakest assumption. That is a legitimate internal dependency: LA-SSM's block-diagonal mixer uses pseudo-phase boundaries to set local window sizes, and if PPN misplaces boundaries the local-dynamics benefit degrades. However, that concern presupposes the model is actually processing long full videos; the more load-bearing issue appears earlier in the pipeline. Section V-B explicitly says every surgical video is further downsampled to 100 frames. If that is literal, the model never sees a full surgical video, so the central novelty claim about full-length holistic processing and linear scaling on long sequences is not supported by the experiments. The figures and text claiming dependencies across hundreds of timesteps cannot be reconciled with a 100-frame input, making this an internal inconsistency, not just a disagreement with convention. The strongest empirical claim, outperforming SOTA by large margins, would still stand as a benchmark result, but it would be for a different, more restricted problem than the one advertised. A code release or a simple dataloader log would settle the ambiguity immediately. If the actual T is the full 1-fps video length, the concern disappears and the paper's claims are much better supported. If T is indeed 100, the authors should either rerun on full-length sequences or explicitly reframe the contribution as short-clip recognition. The PPN issue remains worth an ablation, but it is not the single most load-bearing condition for the central claim. Verdict should remain conditional: acceptance should require resolving this sequence-length ambiguity, preferably by releasing code and/or reporting results at full video length.","tokens_in":15270,"tokens_out":4965,"duration_ms":60963,"concrete_test":"Inspect the released code's data loader and model forward pass. For a representative Cholec80 video (e.g., Video67), log the temporal dimension entering the GR-SSM and LA-SSM layers; if it equals 100 for every video, the full-video claim is unsubstantiated. Then rerun the published splits with (a) the fixed 100-frame subsample and (b) the full 1-fps sequence of the actual video duration; compare Table I-III metrics. Additionally, reproduce Fig. 3's t=300 row from a 100-frame input to check whether the visualization is even possible; if not, the reported long-dependency evidence does not match the stated experimental configuration.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that HID-SSM exploits SSM linear scaling to make decisions on full-length surgical videos. Section V-B says: \"we first uniformly downsample the video to 1 fps and feed the resulting sequence of frames into the visual feature extractor, which further downsample every surgical video to 100 frames.\" A fixed 100-frame temporal input is not a full video; it is a short subsample of each surgery. If this is what the experiments used, the programmatic contribution labeled \"holistic,\" \"full-length,\" and \"long-term dependencies over hundreds of timesteps\" is not actually tested. The description is also internally inconsistent with Fig. 3, which shows matrix-mixer rows at t=300 and claims retrieval of information from \"hundreds of timesteps earlier\"; such rows cannot exist if T=100. The reported SOTA numbers in Tables I-III may therefore be for 100-frame clips rather than complete surgeries, making comparisons with prior work that use longer sequences not apples-to-apples, and the +12.9% Heichole margin would not be evidence about full-video modeling. The PPN windowing concern raised by the reader is real but secondary: even a perfect PPN cannot repair the fact that the input length is capped. This is not an allegation of misconduct; the text is genuinely ambiguous and no code is provided, so the load-bearing experimental condition cannot be verified from the manuscript alone.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript proposes HID-SSM, a hierarchical input-dependent state space model for surgical phase recognition, with the stated goal of making decisions on full-length surgical videos at linear computational cost. The method consists of a temporally consistent visual feature extractor (Swin backbone pre-trained with a shallow SSM head), a local-aggregation SSM block whose windowing is driven by a phase proposal network, a global-relation SSM block, and a hybrid discrete-continuous supervision scheme. Experiments are reported on Cholec80, MICCAI2016, and Heichole, claiming state-of-the-art accuracy with large margins (+2.8%, +4.3%, +12.9%). The paper also provides qualitative visualizations of the learned matrix mixers and of the input-dependent timescale dynamics. The central empirical claim depends on an experimental protocol whose description is internally inconsistent: Section V-B states that every surgical video is downsampled to 100 frames, which contradicts the full-video framing and the qualitative analysis showing timestep 300.","tokens_in":15626,"tokens_out":3506,"duration_ms":39958,"significance":"If the central claim is substantiated, the paper would be a meaningful advance: it adapts selective state space models to surgical phase recognition, combines local and global temporal modeling in a hierarchical way, and provides a clear matrix-mixer interpretation plus a temporal-activation analysis. The empirical gains over transformer-based baselines are large, particularly on Heichole. The manuscript also honestly discloses a non-determinism limitation in Section VI. However, the significance is contingent on the experimental protocol being verifiable and on the full-video claim being tested; the current text leaves the central condition ambiguous, and the claimed superiority cannot be assessed without resolving this ambiguity and without code or error bars for key results. The architectural novelty is moderate—the components are combinations of known SSM ideas—but the application and the reported results could still be valuable if reproducible.","major_comments":[{"comment":"The text states: \"we first uniformly downsample the video to 1 fps and feed the resulting sequence of frames into the visual feature extractor, which further downsample every surgical video to 100 frames.\" This directly contradicts the abstract and introduction's claim that the model enables \"decision making on full-length videos,\" and it is inconsistent with Section V-C2 and Figure 3, which analyze a row at timestep t=300 and claim retrieval of information from \"hundreds of timesteps earlier.\" A fixed 100-frame input is not a full surgical video; it is a short subsample. If this description is accurate, the reported SOTA numbers in Tables I-III are for 100-frame clips, not complete surgeries, and the comparisons with prior work that process longer sequences are not apples-to-apples. The load-bearing experimental condition must be clarified and, if necessary, the experiments rerun on full-length sequences to support the paper's central claim.","section":"§V-B, Implementation Details"},{"comment":"The window size for each LA-SSM block is set to the length of the pseudo-phase predicted by the PPN, which is trained on the same ground-truth phase labels used for final supervision. This creates a self-referential dependency: if the PPN misplaces boundaries, the local aggregation will mix frames from different phases and the claimed local-dynamics advantage degrades. The manuscript does not report PPN accuracy, nor does it provide an ablation with fixed windows, oracle windows, or a sensitivity analysis of boundary errors. This is a load-bearing component of the architecture, and its reliability is not demonstrated.","section":"§IV-C, Phase Proposal Network and LA-SSM"},{"comment":"Section VI discloses that non-deterministic ID-SSM behavior causes performance fluctuations of around 1% in the final model, yet Table III (Heichole) reports only point estimates without error bars or confidence intervals. Given that the claimed improvement over MuST is +12.9%, the absence of any variance measure for the primary Heichole result makes it impossible to judge whether the improvement is significant relative to the disclosed 1% fluctuation. At minimum, multiple-seed results with standard deviations should be reported for all tables, and the discussion should address the interaction between the disclosed randomness and the reported margins.","section":"§VI, Limitation and Table III"}],"minor_comments":[{"comment":"The text refers to \"Surgformer [29]\" but reference [29] is LoViT; Surgformer has reference [7]. The in-text citation should be corrected to avoid confusion.","section":"§V-C1 and References"},{"comment":"Equation (1) writes x'(t) = A x(t-1) + B u(t), which is not a standard continuous ODE (the derivative should depend on x(t), not x(t-1)). This is likely a typo but should be fixed.","section":"§III, Equation (1)"},{"comment":"The column header \"Architechture\" should be spelled \"Architecture.\" Similar typos appear elsewhere (e.g., \"publically\" in the abstract).","section":"Table I header"},{"comment":"The text says \"split the dataset into 40 videos for training, 8 videos for validation, and 32 videos for testing\" and then \"The final performance is reported on all 40 evaluation videos.\" This is confusing because the 40 evaluation videos include both validation and test; please clarify the exact evaluation protocol.","section":"§V-A, Cholec80"},{"comment":"No ablation studies are provided for the three claimed components (temporally consistent feature extractor, LA-SSM, GR-SSM, or the dual-supervision loss). Adding even a simple ablation table would help attribute the reported gains and is standard for a methods paper.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The central issue is the ambiguity in §V-B about the 100-frame downsampling. If the experiments indeed used only 100 frames per video, the paper's main claim of full-video holistic modeling is unsupported, and the comparison with prior work is not meaningful. Even if this is a typo, the lack of code and the self-referential PPN-based windowing require careful verification. I would ask the authors to clarify the protocol, provide code or detailed reproducibility materials, add variance estimates, and ideally run a full-length experiment or provide evidence that the 100-frame subsample does not affect the conclusions. The paper is potentially interesting but currently not verifiable as written."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a look because it combines things I haven't seen together: hierarchical local-aggregation and global-relation input-dependent SSM blocks, a phase proposal network that sets local window sizes from pseudo-phase boundaries, and a discrete-plus-continuous supervision scheme with a phase-progress head. The matrix-mixer visualizations are a nice way to show long-range attention without formal proofs. The reported margins over SR-Mamba and transformer baselines are large, and if the experimental setup is what the text claims, this is a solid subfield advance.\n\nThe soft spots are real, though. Section V-B says the visual feature extractor \"further downsample[s] every surgical video to 100 frames.\" That directly contradicts the paper's core claim of full-video processing, and it also contradicts Figure 3, which shows timestep t=300 in the matrix-mixer panels. If the experiments actually used a fixed 100-frame input, then the \"holistic\" and \"long-term dependencies\" language is not supported by the data, and the comparison to prior work becomes apples-to-oranges. This is the load-bearing ambiguity; it has to be cleared up before the numbers mean what the abstract says they mean. The PPN circularity is secondary but worth noting: the LA-SSM windows come from pseudo-phase predictions trained on the same ground-truth labels, and there is no ablation showing how sensitive performance is to PPN boundary errors.\n\nOther issues are more standard: no code, no ablations on the loss mixing weight alpha or block depths, and no error bars on Heichole. The limitation section is honest about nondeterminism in the ID-SSM backward pass (around 1% fluctuation), which is fine and actually counts in the paper's favor. The citation pattern looks fair, including the SR-Mamba baseline.\n\nMy bottom line: the architecture and the qualitative analysis are strong enough to deserve referee time, but the 100-frame sentence is not a minor typo. If the experiments really are clip-based, the central contribution changes shape. I'd send it to review with a request for code and an explicit statement of the temporal input length; this is the kind of paper where a single clarifying sentence plus code release could make the difference between a solid paper and a misleading one.","headline":"The architecture is genuinely novel and the results are striking, but the implementation section's '100 frames' statement directly contradicts the full-video framing and must be resolved before the headline claims can be trusted.","tokens_in":16142,"tokens_out":1844,"would_cite":false,"duration_ms":24481,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes a hierarchical input-dependent state space model that processes entire surgical videos and reports new state-of-the-art phase recognition on Cholec80, MICCAI2016, and Heichole.","keywords":["surgical phase recognition","state space models","long video understanding","hierarchical temporal modeling","input-dependent timescale","local-global dynamics","discrete-continuous supervision"],"falsifier":"Run HID-SSM with the PPN's pseudo-phase boundaries replaced by random splits of the same average length; if accuracy and Jaccard do not drop substantially on Cholec80, the claimed local-dynamics advantage from learned windowing is not doing the work attributed to it. Alternatively, feed oracle ground-truth phase boundaries to the LA-SSM windows and check whether performance improves beyond PPN windows.","tokens_in":1649,"feed_emoji":"🏥","tokens_out":1507,"duration_ms":62786,"temperature":0.7,"pith_summary":"The paper argues that surgical phase recognition can be done on the whole video at once rather than on clips, by replacing quadratic self-attention with a hierarchical input-dependent state space model that scales linearly with sequence length. It claims this design captures both fine-grained local surgical actions and global workflow structure, and that the resulting HID-SSM outperforms previous state-of-the-art methods by +2.8% accuracy on Cholec80, +4.3% on MICCAI2016, and +12.9% F1 on Heichole. A sympathetic reader would take the central claim to be that linear-complexity selective state space models, combined with temporally consistent visual features and local-global hierarchical blocks, are the right tool for long surgical video understanding.","feed_headline":"SSM reads full surgical videos, sets three phase-recognition records","feed_subtitle":"Hierarchical local-global state space model lifts accuracy by up to 12.9 points over prior methods.","key_machinery":"The central object is the Hierarchical Input-Dependent State Space Model, built from two block types: local-aggregation SSM blocks (LA-SSM) and global-relation SSM blocks (GR-SSM). Each block discretizes an input-dependent ordinary differential equation with a per-frame timescale $\\Delta_t$, so the state transition matrix $\\mathbf{A}_t$ and input matrix $\\mathbf{B}_t$ depend on the current frame; the LA-SSM block uses a block-diagonal matrix mixer to aggregate local dynamics inside pseudo-phase windows proposed by a Phase Proposal Network, while the GR-SSM block uses the full matrix mixer to relate every frame to every other frame at linear cost. The temporal selective activation property, seen when large $\\Delta_t$ resets the state and small $\\Delta_t$ preserves it, lets the model emphasize or ignore frames and is visualized as sharp $\\Delta_t$ changes at phase boundaries.","core_discovery":"The central claim is that a Hierarchical Input-Dependent State Space Model (HID-SSM) can process full-length surgical videos and predict the phase of every frame better than transformer- and CNN-based alternatives. The model stacks local-aggregation SSM blocks, which operate within windows proposed by a Phase Proposal Network, on top of global-relation SSM blocks that mix information across the whole sequence, and it is trained with both discrete phase labels and continuous phase-progress targets. In the paper's experiments, the causal variant reaches 94.5% accuracy on Cholec80 and the contextual variant reaches 96.2%, with corresponding gains on MICCAI2016 and Heichole, which the paper reports as new state-of-the-art results.","pith_inferences":["If the margin on Heichole (+12.9 F1) generalizes, the largest gains come in low-data settings, suggesting the hierarchical SSM's inductive bias matters most where transformer training data is scarce; this is an editorial inference, not a paper claim.","The same architecture could transfer to other long-video tasks with natural segment structure, such as action segmentation or procedural video understanding, where pseudo-phase proposals could come from any coarse temporal segmenter.","A direct ablation that corrupts or replaces PPN pseudo-phase boundaries would isolate how much of the gain comes from learned windowing versus the SSM blocks themselves; this is testable and the paper does not report it.","Because the contextual (bidirectional) HID-SSM substantially outperforms the causal version on Heichole, online causal deployment may require accepting a tradeoff; the paper reports both so a practitioner could choose based on latency constraints."],"forward_implications":["Full-video processing at linear cost becomes practical, so surgical phase recognition need not rely on fixed-size windowing or multi-GPU transformer training.","The hierarchical local-global design gives an internal segmentation (pseudo-phases) that makes local aggregation semantically coherent, improving fine-grained phase-transition recognition.","Training with continuous phase-progress supervision alongside discrete labels regularizes the SSM and improves transition awareness.","Visualizations of the matrix mixer and $\\Delta_t$ offer interpretability: the model retrieves information from hundreds of frames back and resets its state at phase jumps.","The reported margins (+2.8%, +4.3%, +12.9%) set a new state of the art on the three datasets under the relaxed and unrelaxed protocols used."],"supporting_citations":[{"why":"Supplies the input-dependent selective state space model and hardware-aware algorithm used for discretized recurrent computation.","marker":"[21]"},{"why":"Supplies the matrix mixer form used to express and visualize both LA-SSM and GR-SSM blocks.","marker":"[22]"},{"why":"Supplies the bidirectional extension of the matrix mixer used in the contextual HID-SSM variant.","marker":"[23]"},{"why":"Transformer baseline with fixed-size windowing that HID-SSM must surpass on Cholec80 and MICCAI2016.","marker":"[6]"},{"why":"Strong transformer baseline on Cholec80 whose accuracy and Jaccard scores are compared against HID-SSM.","marker":"[7]"},{"why":"Multi-stage temporal convolutional network baseline compared on all three datasets.","marker":"[12]"},{"why":"Linear state space baseline that provides a direct comparison for the SSM-based approach.","marker":"[9]"},{"why":"Earlier SSM application to surgical phase recognition that the proposed method extends.","marker":"[24]"},{"why":"Provides the Cholec80 dataset and its annotation protocol used for training and evaluation.","marker":"[3]"},{"why":"Provides the Heichole dataset whose complex phase interleaving stresses long-range modeling.","marker":"[34]"}],"fun_headline_variants":["State space model decodes full surgical videos end-to-end","Hierarchical SSM lifts surgical phase accuracy up to 12.9 pts","Input-dependent SSM sets three surgical phase recognition records","SSM reads full surgery videos, sets three phase-recognition records"],"cache_read_input_tokens":18176,"weakest_assumption_plain":"The method assumes the Phase Proposal Network finds pseudo-phase boundaries accurate enough that, when each LA-SSM block's window is set to a predicted pseudo-phase, frames from different surgical phases are not mixed together.","fun_headline_variants_meta":{"raw":{"variants":["State space model decodes full surgical videos end-to-end","Hierarchical SSM lifts surgical phase accuracy up to 12.9 pts","Input-dependent SSM sets three surgical phase recognition records","SSM reads full surgery videos, sets three phase-recognition records"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001344,"raw_usage":{"total_tokens":5450,"prompt_tokens":922,"completion_tokens":4528,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":4456}},"tokens_in":538,"tokens_out":4528,"duration_ms":37437,"temperature":1.0,"reasoning_tokens":4456,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:26:30.124614+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run HID-SSM with the PPN's pseudo-phase boundaries replaced by random splits of the same average length; if accuracy and Jaccard do not drop substantially on Cholec80, the claimed local-dynamics advantage from learned windowing is not doing the work attributed to it. Alternatively, feed oracle ground-truth phase boundaries to the LA-SSM windows and check whether performance improves beyond PPN windows.","supporting_citations":[{"cited_title":"Mamba: Linear-time sequence modeling with selective state spaces,","cited_arxiv_id":null,"evidence_quote":"Supplies the input-dependent selective state space model and hardware-aware algorithm used for discretized recurrent computation."},{"cited_title":"Transformers are ssms: Generalized models and efficient algorithms through structured state space duality,","cited_arxiv_id":null,"evidence_quote":"Supplies the matrix mixer form used to express and visualize both LA-SSM and GR-SSM blocks."},{"cited_title":"Hydra: Bidirectional state space models through generalized matrix mixers,","cited_arxiv_id":null,"evidence_quote":"Supplies the bidirectional extension of the matrix mixer used in the contextual HID-SSM variant."},{"cited_title":"Trans-svnet: Accurate phase recognition from surgical videos via hybrid embedding aggregation transformer,","cited_arxiv_id":null,"evidence_quote":"Transformer baseline with fixed-size windowing that HID-SSM must surpass on Cholec80 and MICCAI2016."},{"cited_title":"Surgformer: Surgical transformer with hierarchical temporal attention for surgical phase recognition,","cited_arxiv_id":null,"evidence_quote":"Strong transformer baseline on Cholec80 whose accuracy and Jaccard scores are compared against HID-SSM."},{"cited_title":"Czempiel, M","cited_arxiv_id":null,"evidence_quote":"Multi-stage temporal convolutional network baseline compared on all three datasets."},{"cited_title":"Simplified state space layers for sequence modeling,","cited_arxiv_id":null,"evidence_quote":"Linear state space baseline that provides a direct comparison for the SSM-based approach."},{"cited_title":"Sr-mamba: Effective surgical phase recognition with state space model,","cited_arxiv_id":null,"evidence_quote":"Earlier SSM application to surgical phase recognition that the proposed method extends."},{"cited_title":"Endonet: A deep architecture for recognition tasks on laparoscopic videos,","cited_arxiv_id":null,"evidence_quote":"Provides the Cholec80 dataset and its annotation protocol used for training and evaluation."},{"cited_title":"Comparative validation of machine learning algorithms for surgical workflow and skill analysis with the heichole benchmark,","cited_arxiv_id":null,"evidence_quote":"Provides the Heichole dataset whose complex phase interleaving stresses long-range modeling."}],"review_version":1}