{"id":"e2970e15-04c7-4e6d-a31c-bff1d765d8e6","arxiv_id":"2505.24287","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"EgoExOR is a new multimodal, multi-perspective OR dataset with 84,553 annotated frames, plus a benchmark showing that fusing egocentric and exocentric signals improves surgical scene graph generation.","lead":"The authors release EgoExOR, a new operating room dataset that synchronizes first-person glasses footage (RGB, gaze, hand tracking, audio) with room-level RGB-D cameras and ultrasound screen captures during simulated spine procedures. It is the first OR benchmark to pair ego and exo perspectives with dense scene graph annotations, and it shows that combining both views improves automated surgical scene understanding.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Synchronization is asserted, not demonstrated: a single clapper at take start plus 'no drift observed' cannot support frame-level alignment across independent Aria glasses, Kinects, and HDMI capture over 94-minute sessions.","rationale":"The reader's verdict is CONDITIONAL, with the weakest assumption being the same clapper-based synchronization in Section 3.2. I agree: the paper's only evidence for frame-level alignment across independent recording devices is a single clapper at take start and a subjective 'no drift observed.' The data is public, so the proposed drift audit can be run. If drift exceeds half a frame, the dataset's core promise and every reported F1 number lose their meaning. I considered whether the internal inconsistency in Table 4 (where 'Ego Only' retains ultrasound/audio/point cloud and 'Exo Only' includes exocentric RGB plus those same non-RGB modalities) is more load-bearing. It does undermine the specific ablation conclusion about perspective complementarity, but the central claim of the paper is the dataset itself and the benchmark, not the clean isolation of ego vs. exo contributions. That flaw is correctable without invalidating the dataset. Similarly, the missing error bars and inter-annotator agreement are quality issues, not fundamental threats. The synchronization issue, by contrast, is a necessary condition for every downstream use of the dataset. Hence I keep the reader's CONDITIONAL verdict unchanged, but sharpen the condition: the authors must provide a quantitative drift analysis or release the synchronization metadata before the benchmark claims can be accepted.","tokens_in":14737,"tokens_out":8307,"duration_ms":106409,"concrete_test":"Use the public HDF5 archive. For each of at least three takes (the longest UI take, the longest MISS take, and one short take), detect a sharp audio transient in each Aria microphone track and the corresponding visual transient (e.g., the clapper, or a later instrument drop) in each camera stream. Compute the per-stream offset relative to the Azure Kinect master clock at take start and again at a second event near the end of the take. If the end-of-take offset differs from the start-of-take offset by more than 33 ms (half a 15 FPS frame) for any stream, the 'no drift' statement is falsified and frame-level fusion is unsupported. If all offsets stay within 33 ms, the synchronization concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 states that Azure Kinects are hardware-synchronized, but all other streams (Aria glasses, HDMI ultrasound capture) are aligned manually after a single clapper at the beginning of each take. The authors write 'We did not observe any drift in the streams during the duration recordings' with no quantitative synchronization error reported. At 15 FPS, one frame is 66.7 ms; even a relative clock drift of 0.01% (100 ppm, typical for free-running consumer oscillators) accumulates to 0.56 s over a 94-minute session, i.e., ~8 frames. A single clapper event cannot correct for such drift, and visual inspection only catches errors once they are already many frames. Since every dataset timepoint is defined by a frame index, any multi-frame misalignment between the egocentric RGB, gaze, hand tracking, exocentric RGB-D, and ultrasound streams means the scene graph supervision is attached to content that is not temporally coincident across modalities. The reported benchmark F1 differences (Table 3) then reflect synchronization artifacts rather than true multimodal fusion. The dataset's central value proposition is precisely its synchronized multi-perspective acquisition, so this unvalidated assumption is load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EgoExOR, a new operating-room dataset recorded at 15 FPS over 94 minutes (84,553 frames) from two emulated spine procedures: Ultrasound-Guided Injection (UI) and Minimally Invasive Spine Surgery (MISS). The dataset combines egocentric RGB, gaze, hand tracking, and audio from Project Aria glasses (worn by different roles, plus microscope and OR-light viewpoints) with exocentric RGB-D from five ceiling-mounted Azure Kinects, ultrasound screen captures, and per-frame colored point clouds. It provides dense scene graph annotations with 36 entity classes, 22 relation classes, and 568,235 triplets. The paper also defines a surgical scene graph generation benchmark, adapts two existing models (ORacle and MM2SG), and proposes a new dual-branch baseline that fuses egocentric and exocentric modalities. Reported macro F1 scores are 0.63 for ORacle, 0.67 for MM2SG, and 0.72 for the proposed model, with an ablation in the appendix showing egocentric-only (0.68) and exocentric-only (0.42) variants.","tokens_in":14948,"tokens_out":7441,"duration_ms":88651,"significance":"If the synchronization and annotation claims hold, EgoExOR is a potentially valuable and genuinely novel resource: it appears to be the first OR dataset to combine synchronized multi-view egocentric and exocentric recordings with gaze, hand pose, screen capture, and dense scene graph annotations, and it is publicly released with a documented HDF5 structure, missing-data conventions, and pre-defined train/val/test splits. The benchmark is a reasonable starting point for surgical scene graph generation, and the per-predicate results in the appendix give useful diagnostic information. The machine-checkable release format, clearly stated sensor setup, and explicit ablation of ego vs exo inputs are strengths. The proposed dual-branch model is a modest baseline contribution whose main interest is in showing that the new modalities can be consumed; the dataset itself, not the model, is the primary contribution.","major_comments":[{"comment":"The synchronization of the Aria glasses, ultrasound HDMI capture, and Azure Kinects is asserted but not quantitatively demonstrated. The text states that a clapper at the beginning of each take was used and that 'We did not observe any drift in the streams during the duration recordings,' but no synchronization error is reported. At 15 FPS one frame is 66.7 ms, and even a modest relative clock drift of 100 ppm accumulates to about 0.56 s (roughly 8 frames) over a 94-minute session. Because every scene graph annotation is attached to a frame index and the benchmark F1 scores in Table 3 presuppose that all modalities are temporally coincident, the core value proposition of a synchronized ego-exo dataset is not yet established. In addition, Appendix A.2 says acquisition was continuous over nine raw sessions and takes were segmented post hoc, which is hard to reconcile with a clapper at the start of each take. Please provide per-take or per-session quantitative synchronization validation (e.g., maximum frame offset, drift rate, or cross-modal event alignment error) and clarify the actual calibration protocol.","section":"Section 3.2, Section 4.1, Appendix A.2"},{"comment":"All macro F1 scores are reported from a single training run with no error bars, confidence intervals, or significance tests. The headline gap between the proposed model (0.72) and MM2SG (0.67) is only 0.05, and the ablation in Table 4 shows a 0.04 gap between the egocentric-only variant (0.68) and the full model. Without multiple seeds or statistical testing, these differences could be within training noise. Please report mean and standard deviation over at least three independent training runs (or bootstrap confidence intervals) for all rows in Tables 3 and 4, and state whether the observed gaps are statistically significant.","section":"Table 3 and Table 4"},{"comment":"The annotation process is described as one trained annotator labeling each frame and a second annotator verifying the output, but no inter-annotator agreement metric is reported. Given that the dataset's primary supervision signal is 568,235 scene graph triplets, a sample-based agreement measure (e.g., triplet-level Cohen's kappa or pairwise F1 between two independent annotators) is necessary to establish annotation reliability and to calibrate the achievable performance ceiling for the benchmark.","section":"Section 4.3"},{"comment":"The supplementary material defines conventions for absent signals (zero-filled RGB and audio, NaN hand keypoints, invalid gaze as (-1,-1), empty point clouds) but never quantifies how often each modality is missing across the 41 takes. If gaze, hand tracking, point clouds, or ultrasound are unavailable for a substantial fraction of timepoints, the comparison in Table 3 between models that consume those modalities and those that do not is confounded by the imputation strategy. Please report per-modality coverage statistics (e.g., the percentage of frames with valid data per take and overall).","section":"Appendix A.2 (Missing-data handling)"},{"comment":"The contribution of the dual-branch architecture is not isolated from the contribution of the additional modalities. MM2SG does not use gaze or hand data, so the 0.05 F1 gain achieved by the proposed model could be entirely due to these extra input signals rather than to the perspective-specific branch structure. To support the architectural claim, add an ablation that feeds all seven modalities through a single shared encoder (e.g., an ORacle/MM2SG-style unified stream with the same token budget) and compare it with the dual-branch variant.","section":"Section 5, Figure 2, Table 3"}],"minor_comments":[{"comment":"The caption uses 'EgoExoR' but the dataset is called 'EgoExOR'; please make the spelling consistent.","section":"Figure 1 caption"},{"comment":"'LLaV A-7B' contains an erroneous space; it should read 'LLaVA-7B'.","section":"Section 5, Implementation Details"},{"comment":"'ressource' should be 'resource'.","section":"Section 6, Limitations"},{"comment":"The phrase 'sub-millimeter accuracy (2D pixel + depth)' for the Aria eye-tracking system is not accompanied by a reference or a specification of the measurement conditions; please cite the Project Aria documentation or state the accuracy more cautiously.","section":"Section 3.2"},{"comment":"The checkmark alignment in the table is ambiguous, especially for the 'Ego Only' row, which appears to show five checkmarks but does not clearly indicate which columns are selected; please render the table so that each checked modality is unambiguously visible.","section":"Table 4"},{"comment":"The sentence 'While using only egocentric input performs much better' is grammatically incomplete and should be rephrased, for example, 'While using only egocentric input performs much better than exocentric-only input, the best performance is achieved when all inputs are used together.'","section":"Appendix A.1"},{"comment":"Reference [5] is listed as 'Egosurgery-tool' but the text refers to 'EgoSurgery'; please standardize the capitalization and verify the reference formatting.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is from the same group that authored the two baselines (ORacle and MM2SG), which may create an incentive to present the new baseline favorably, but the evaluation protocol itself appears standard and the comparisons use fixed splits. The dataset's public release, documented format, and missing-data conventions are strong points. The main risk is the unvalidated synchronization; if the authors can supply quantitative sync error measurements and multi-seed results, the paper would be substantially stronger. The 'first OR dataset' claim in Table 1 depends on the completeness of the comparison; I did not independently verify all listed dataset features but found no immediate misrepresentation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"EgoExOR is the first OR dataset to combine egocentric and exocentric views with gaze, hand pose, and scene graphs. That claim holds up against the prior work in Table 1. The dataset itself is a real artifact: public code, data, detailed acquisition and annotation descriptions, and 84k frames of simulated spine procedures. The combination is new, not a re-packaging.\n\nThe benchmarking is the soft part. The synchronization of the Aria glasses, Kinects, and HDMI capture rests on a single clapper per take and \"we did not observe drift.\" That is asserted, not measured. At 15 FPS, even 100 ppm clock drift gives several frames of error over a 94-minute session. If the streams are not actually frame-aligned, the F1 numbers lose meaning. Reviewers should ask for a quantitative sync validation, or at least a description of how drift was corrected. This is fixable, but it is load-bearing.\n\nSecond, all F1 scores are single runs with no variance. The 0.72 vs 0.67 gap could be within noise. Third, the ablation in Table 4 is mislabeled: \"Ego Only\" includes ultrasound, audio, point cloud, gaze, and hand; \"Exo Only\" includes ultrasound and audio. The rows don't match the names, so the ablation doesn't cleanly isolate the ego/exo contribution. Easy fix, but it needs to happen.\n\nThe baselines and the new model come from the same lab, so the comparison is not independent. That is a reason to be cautious, not a flaw by itself.\n\nOverall, the dataset is a useful contribution and the paper deserves peer review. The limitations listed by the authors (simulated, single OR layout) are honest. I'd accept it for review, but with a request for sync validation and error bars before the numbers are relied on. A serious referee could also ask for annotation consistency checks.\n\nFor someone in surgical vision or ego-exo activity understanding, this is worth citing. For a general reading group, maybe—the methods are standard, but the dataset is the point.","headline":"A genuinely new OR dataset worth engaging; the benchmark numbers need sync validation and error bars before they can be trusted.","tokens_in":15545,"tokens_out":3326,"would_cite":true,"duration_ms":40812,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces EgoExOR, a surgical dataset that synchronizes egocentric wearable streams with exocentric room cameras, and reports a dual-branch scene-graph model reaching 0.72 macro F1.","keywords":["egocentric vision","exocentric vision","operating room dataset","scene graph generation","surgical activity understanding","gaze tracking","hand pose estimation","multimodal fusion"],"falsifier":"Open the released HDF5 files and measure the inter-stream delay for each take: detect the clapper event in each camera's audio and video, then track a visible event across the full take to see whether stream offsets drift by more than one frame, about 67 ms at 15 FPS. Any such drift would break the frame-level correspondence between scene graph labels and audiovisual content, and the reported F1 scores would need to be recomputed on correctly aligned frames.","tokens_in":14529,"feed_emoji":"🩺","tokens_out":7244,"duration_ms":84639,"temperature":0.7,"pith_summary":"EgoExOR is presented as the first operating-room dataset to synchronize multi-view first-person and third-person recordings: three wearable glasses (RGB, gaze, hand pose, audio), five overhead RGB-D cameras, and an ultrasound screen capture, recorded over 94 minutes of two emulated spine procedures. The paper's central claim is that this combination matters: a dual-branch model that keeps egocentric and exocentric cues separate reaches a macro F1 of 0.72 on surgical scene graph generation, outperforming the adapted MM2SG (0.67) and ORacle (0.63) baselines. An ablation shows exocentric-only input falls to 0.42 and egocentric-only to 0.68, supporting the claim that the two perspectives carry complementary information. The dataset, with 36 entity classes, 22 relation classes, and 568,235 triplets across 84,553 frames, is offered as a benchmark resource for OR perception.","feed_headline":"First OR dataset to fuse ego and room views hits F1 0.72","feed_subtitle":"A dual-branch model beats prior multimodal baselines by keeping surgeon-view and room-view signals separate.","key_machinery":"The carrier of the argument is the synchronized EgoExOR capture itself together with the dual-branch fusion model. The dataset's annotations, covering 36 entity classes and 22 relation classes, give the benchmark its target, while the model's dual-branch design is the mechanism that tests the dataset's value: each branch applies a 2-layer transformer to its modalities, pools to N embeddings, concatenates them, and lets a large vision-language model predict triplets, so fine-grained egocentric cues such as gaze and hand pose are kept separate from global exocentric cues instead of being mixed in a single encoder.","core_discovery":"The paper establishes EgoExOR as a dataset that unites perspectives no prior OR dataset combined: egocentric RGB, gaze, hand pose, and audio from wearable glasses, plus exocentric RGB-D, ultrasound screen capture, and a fused point cloud, all synchronized at 15 FPS across 41 takes of two simulated spine procedures. On top of this multimodal capture, the authors propose a dual-branch baseline for surgical scene graph generation: an egocentric branch encodes first-person RGB, hand pose, and gaze, an exocentric branch encodes RGB-D, ultrasound, audio, and point clouds, and the fused embeddings are fed to a vision-language model that autoregressively emits (subject, predicate, object) triplets. The reported result is a macro F1 of 0.72 overall (0.79 on ultrasound-guided injection, 0.68 on minimally invasive spine surgery), beating adapted MM2SG (0.67) and ORacle (0.63), with the ablation showing that egocentric-only input reaches 0.68 while exocentric-only input drops to 0.42.","pith_inferences":["If the egocentric branch carries most of the fine-grained predicate signal, a wearable-only model might support real-time assistance in constrained settings, trading global context for lower sensor overhead; the paper does not test this.","Because synchronization relies on a clapper with no reported error bound, a quantitative synchronization-error audit would make the reported F1 numbers reproducible and comparable with future ego-exo OR datasets.","The simulated, scripted procedures mean the reported F1 may not transfer directly to live operating rooms; measuring the domain gap on real surgical recordings would be a natural extension.","The class-imbalance pattern suggests that dataset growth should target rare predicates such as cutting and anaesthetising rather than simply adding more frames of common interactions."],"forward_implications":["Adding egocentric signals to exocentric OR models is not incremental: in the reported benchmark, the full dual-branch model outperforms both single-perspective variants, and exocentric-only performance is far lower.","Fine-grained tool-hand predicates such as injecting, aspirating, and controlling score well when gaze and hand pose are available, indicating that wearable sensing carries a substantial part of the interaction signal.","Low-frequency predicates remain the failure mode: cutting and anaesthetising score far below the macro average, pointing to class imbalance as the next bottleneck for OR scene graph generation.","The 84,553 annotated timepoints give a new training and evaluation resource for multi-perspective scene graph generation and, more broadly, for human-centric OR perception tasks.","More modalities lead to better scene graph prediction in the reported experiments, supporting the design decision to include ultrasound, audio, point clouds, gaze, and hand pose in a synchronized benchmark."],"supporting_citations":[{"why":"Introduced semantic scene graphs for operating-room modeling, the annotation and evaluation scheme that EgoExOR adopts.","marker":"[2]"},{"why":"Provides the large multimodal OR dataset and the MM2SG baseline that EgoExOR compares against and extends.","marker":"[3]"},{"why":"Supplies the only prior egocentric surgical dataset, establishing the gap in first-person OR perception that EgoExOR fills.","marker":"[4]"},{"why":"Demonstrates the ego-exo multimodal capture paradigm for skilled activities that EgoExOR transplants to the operating room.","marker":"[15]"},{"why":"Describes the Project Aria glasses used to capture egocentric RGB, gaze, hand tracking, and audio.","marker":"[29]"},{"why":"Presents the ORacle baseline for surgical scene graph generation that EgoExOR adapts and outperforms.","marker":"[40]"},{"why":"Provides the LLaVA vision-language model used as the starting point for all benchmark models.","marker":"[41]"}],"fun_headline_variants":["EgoExOR: first OR dataset fusing ego and exo views, scene graph F1 0.72","Dual-branch model fuses glasses and room cameras for OR scene graph F1 0.72","Fusing ego and exo views lifts OR scene graph F1 to 0.72","OR dataset with ego glasses + room RGB-D hits F1 0.72 in scene graphs","EgoExOR combines surgeon and room views, beats baselines with F1 0.72"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the assumption that the clapper-based manual synchronization keeps all streams aligned at frame level for all 41 takes; the paper states no drift was observed but reports no quantitative synchronization error.","fun_headline_variants_meta":{"raw":{"variants":["EgoExOR: first OR dataset fusing ego and exo views, scene graph F1 0.72","Dual-branch model fuses glasses and room cameras for OR scene graph F1 0.72","Fusing ego and exo views lifts OR scene graph F1 to 0.72","OR dataset with ego glasses + room RGB-D hits F1 0.72 in scene graphs","EgoExOR combines surgeon and room views, beats baselines with F1 0.72"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000505,"raw_usage":{"total_tokens":2494,"prompt_tokens":1002,"completion_tokens":1492,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":618,"completion_tokens_details":{"reasoning_tokens":1364}},"tokens_in":618,"tokens_out":1492,"duration_ms":11327,"temperature":1.0,"reasoning_tokens":1364,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:26:47.068894+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Open the released HDF5 files and measure the inter-stream delay for each take: detect the clapper event in each camera's audio and video, then track a visible event across the full take to see whether stream offsets drift by more than one frame, about 67 ms at 15 FPS. Any such drift would break the frame-level correspondence between scene graph labels and audiovisual content, and the reported F1 scores would need to be recomputed on correctly aligned frames.","supporting_citations":[{"cited_title":"4d-or: Semantic scene graphs for or domain modeling","cited_arxiv_id":null,"evidence_quote":"Introduced semantic scene graphs for operating-room modeling, the annotation and evaluation scheme that EgoExOR adopts."},{"cited_title":"Mm-or: A large multimodal operating room dataset for semantic understanding of high-intensity surgical environments","cited_arxiv_id":null,"evidence_quote":"Provides the large multimodal OR dataset and the MM2SG baseline that EgoExOR compares against and extends."},{"cited_title":"Egosurgery-phase: A dataset of surgical phase recognition from egocentric open surgery videos","cited_arxiv_id":null,"evidence_quote":"Supplies the only prior egocentric surgical dataset, establishing the gap in first-person OR perception that EgoExOR fills."},{"cited_title":"Project aria: A new tool for egocentric multi-modal ai research, 2023","cited_arxiv_id":null,"evidence_quote":"Describes the Project Aria glasses used to capture egocentric RGB, gaze, hand tracking, and audio."},{"cited_title":"Oracle: Large vision- language models for knowledge-guided holistic or domain modeling","cited_arxiv_id":null,"evidence_quote":"Presents the ORacle baseline for surgical scene graph generation that EgoExOR adapts and outperforms."}],"review_version":1}