HOLa: HoloLens Object Labeling

Johannes Schmidt; Konstantin Holzapfel; Michael Schwimmbeck; Serouj Khajarian; Stefanie Remmele

REVIEW 3 major objections 4 minor 6 references

HOLa: HoloLens Object Labeling

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read HOLa: a single seed point plus a tracking model labels HoloLens surgery video at human-level quality, more than 500 times faster than manual annotation.

desk verdict HOLa is a useful, honest tool paper for HoloLens annotation, but the headline Dice numbers need a random-frame re-evaluation before I'd trust them. read the letter →

arxiv 2412.04945 v2 pith:7O37WP4Q submitted 2024-12-06 cs.CV cs.LG

Michael Schwimmbeck , Serouj Khajarian , Konstantin Holzapfel , Johannes Schmidt , Stefanie Remmele This is my paper · ORCID

classification cs.CVcs.LG

keywords HoloLens2objectlabelingdataannotationSegmentAnythingModelSAM-TrackaugmentedrealitysurgeryvideosegmentationmedicalAR

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces HOLa, a HoloLens 2 recording and labeling pipeline that turns a single center-of-object seed point into pixel-wise masks for every RGB frame in a recording. Its central claim is that this fully automatic annotation reaches human-level quality: in five liver-surgery and phantom experiments the mean Dice score against a human reference lies between 0.875 and 0.982, close to the agreement among human annotators, while annotation throughput rises from about 0.008–0.010 frames per second to 5 frames per second. The authors argue this removes the main bottleneck in collecting training data for AR-guided surgery tracking models. Because the segmentation backbone is a foundation model, the pipeline is not tuned to a specific image appearance and is offered as a general solution for AR object labeling.

What carries the argument

The load-bearing mechanism has three parts: the seed-point-prompted Segment Anything Model (SAM), which turns one click at the frame center into an initial object mask by picking the highest-IoU proposal among three candidates; the DeAOT tracker from SAM-Track, which propagates that mask across the video; and the HL2SS sensor-streaming plugin, which brings the HoloLens RGB camera, depth, point cloud, and poses into the labeling pipeline. The authors replace SAM-Track's 'Segment Everything' mode with the single-seed SAM prompt so that exactly one object is followed. This combination is what lets a single initialization label an entire recording.

What would settle it

Re-run HOLa on the complete, unedited recordings from the five experiments, not on the author-selected 90 frames, and have annotators who were not involved in recording label all frames; if the mean Dice on the full sequences falls below the human inter-rater concordance by more than the small margins reported here, the claim of human-level automatic labeling on HoloLens recordings is refuted.

Watch

Extended reading notes

Core claim

HOLa's core discovery is that a promptable segmentation foundation model combined with a video object tracker can replace frame-by-frame manual labeling on HoloLens 2 recordings without loss of quality. The user points a sphere cursor at the object and speaks a command; SAM is prompted at the first frame with that seed point, the highest-IoU mask among its proposals initializes the DeAOT tracker, and the tracker propagates the mask through all subsequent frames. On the reported five experiments the mean Dice scores of HOLa versus the human reference are 0.982, 0.966, 0.981, 0.925, and 0.875, and on a 10-frame subset the HOLa-versus-human concordance tracks the human-versus-human concordance closely (for example 0.887 versus 0.917 on the most difficult surgery scene). The same experiments show a speedup of more than 500 times relative to manual annotation.

Load-bearing premise

The headline quality numbers rest on the assumption that the 90 frames selected per experiment as best representing the recording are a representative, unbiased sample; because the same person who made the recordings chose those frames and served as the reference annotator, agreement on arbitrary unselected HoloLens frames could be lower.

Editorial extensions

If this is right

Researchers collecting HoloLens training data for organ tracking can generate pixel-wise labels for entire recordings by marking one seed point, cutting annotation labor by orders of magnitude.
The labeling quality on clearly separated organs (Dice above 0.96 in phantoms, 0.925 in the first surgery scene) means the output can serve as training masks with only light quality control.
In low-contrast scenes where an organ blends into surrounding tissue, automatic labels degrade to about 0.875 Dice, so those recordings need human review or additional seed points.
Because the method tracks one object only, complex multi-segment structures require placing extra seed points during quality control, and frames where the object leaves the view are not labeled.
The approach transfers without appearance-specific tuning, so the same pipeline applies to non-medical HoloLens AR labeling tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reported 500x speedup compares automated post-processing on a high-end GPU with manual labeling at a desk; an end-to-end count that includes recording time, seed placement, and quality control would be smaller, though likely still large.
Because the reference annotator was the person who recorded and selected the frames, the human-level scores may partly reflect familiarity with the scenes; an evaluation with independent annotators selecting frames at random would give a stricter estimate.
A natural extension is to feed the synchronized depth stream and point cloud into the tracker; the paper's own examples show boundary errors from shadows and low color contrast, and depth cues are precisely the kind of signal that could correct those.
If the same seed-point-and-track recipe is applied to newer promptable segmentation models, the ranking of results across the five scenes would probably track the model's boundary quality on low-contrast imagery rather than anything specific to HoloLens.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear [Colleague],

Short version: this is a legitimately useful tool paper, and the evaluation is mostly sound, but the headline Dice numbers sit on a non-random frame-selection step that needs rechecking. I'd send it out for review, with a request for a more robust validation.

What's actually new: HOLa is an open-source HoloLens 2 recording and labeling app that wires HL2SS streaming to SAM-Track, modified for single-object, seed-point-prompted tracking. The integration is straightforward—no new architecture—but the package is real. The paper reports five experiments (three phantoms, two in-situ liver surgeries) with Dice between 0.875 and 0.982, and a ~500x speedup over manual labeling. The public code and the clinical data with ethics approval are concrete assets. The authors also state the limitations honestly: single-object only, SAM fails on low contrast, seed placement matters, and human QC is always required. That last point undercuts the 'fully automatic' label in the abstract, but they don't hide it.

Where it gets soft: the 90 frame evaluation. You pick frames 'that best represent the variations during recording'—that's post-hoc selection, not sampling. The authors say no frames were excluded 'in advance,' but that doesn't address frames excluded by the selection itself. The reference rater is the person who recorded the data and placed the seed point, which is a familiarity bias even if two medical experts revised the labels. The 10-frame subset used for inter-rater comparison is also non-random, and the 0.013 Dice difference between 10 and 90 frames only shows two non-random subsets agree with each other. So the reported quality range is probably optimistic for arbitrary frames, especially in low-contrast Experiment 5. This is fixable: re-annotate random or all frames without the 'best represent' filter, or at least report per-frame Dice distributions and failure cases. The speedup is also only for the automatic pass; the human QC pass burns time, though that's a minor framing issue.

The center of the paper—that a SAM-Track pipeline seeded by a HoloLens cursor can produce useful masks—holds up. The weak spot is the external validity of the exact numbers, not the tool itself.

Who it's for: anyone building HoloLens medical-AR datasets who wants to skip manual pixel annotation. It's a solid engineering contribution, and the evaluation is good enough to be a serious referee assignment, not a desk reject. I'd ask for the random-frame re-evaluation before acceptance, but I wouldn't block on anything else.

Referee Report

3 major / 4 minor

Summary. The paper presents HOLa, an integrated Unity/Python application for the Microsoft HoloLens 2 that combines the Segment Anything Model (SAM) with the DeAOT tracker (SAM-Track) to annotate a single object of interest across recorded video frames. The user selects a seed point via an AR sphere cursor and initiates recording by voice command; the labeling mode then produces pixel-wise masks for every frame. The method is evaluated on three phantom experiments and two open-liver-surgery recordings. The authors report mean Dice scores between 0.875 and 0.982, close to human inter-rater concordance on a 10-frame subset, and an annotation speed of 5 fps versus roughly 0.008–0.010 fps for humans, yielding the advertised 'more than 500 times' speedup. The code is publicly available.

Significance. If the reported quality and speed hold, HOLa would be a practically useful tool for building annotated datasets for medical AR tracking research, with a meaningful reduction in manual labeling cost. The paper is, to the authors' knowledge, the first evaluation of SAM-based tracking on HoloLens RGB data, and the integration with a HoloLens recording application is a tangible contribution. There is no obvious circularity in the evaluation: HOLa is not fit to the test data, and the Dice scores compare its outputs with independent human annotations (HA 1). The main strength is the concrete, open-source system that others can reuse. The main weakness is the evaluation protocol: only five short recordings are used, the 90-frame subsets are chosen retrospectively and subjectively by the authors, and the reference annotator is the person who recorded the data. These issues directly affect the credibility of the headline quantitative claims, although they are fixable with additional experiments and analysis.

major comments (3)

[Section 3 (Experiments)] The evaluation protocol does not support the claim that the reported Dice scores are representative of HOLa's performance on typical HoloLens usage. The 90 frames per experiment are selected by the authors 'across the entire sequence that best represent the variations during recording,' and the statement 'No frames were excluded in advance' does not address the fact that frames were excluded post hoc by this subjective selection step. If the selection oversamples easy, high-contrast frames and undersamples tracker drift or boundary-ambiguity frames, the mean Dice scores, especially the 0.875 for Experiment 5, will be optimistically biased. The validity of the central quality claim requires an evaluation on all recorded frames or on a pre-registered random/blinded subset, reported with frame-level distributions and worst-case scores.
[Section 5 (Discussion)] The defense of the evaluation in the Discussion is insufficient. The sentence 'the metrics differ by less than 0.013 Dice compared to the results based on 90 frames, suggesting that the selection is representative for the total set' only compares two non-random subsets (90 vs. 10 frames) that were both hand-picked by the authors; consistency between two subjective selections does not establish representativeness of the full recording. Furthermore, HA 1 is 'the same person who recorded the data,' so the reference annotations may benefit from a familiarity with the scenes that an independent oracle would not have. The authors should provide either a blinded re-annotation study or an all-frame/random-frame evaluation to rule out this bias.
[Table 1 and Section 5] The headline 'more than 500 times' speedup is computed for the automatic labeling pass only, but the Discussion states that 'this will never replace a human cross-check' and that future work will analyze quality control more closely. As stated, the end-to-end labeling workflow includes a human QC step whose cost is not included in the reported 5 fps. The speed claim should be qualified to the automatic pass, or the authors should estimate the full time including QC to support the practical workload-reduction claim.

minor comments (4)

[Abstract] The phrase 'fully automatic single object annotation ... while requiring minimal human participation' is almost contradictory; consider rephrasing to 'automatic labeling after a single user-provided seed point' to match the actual workflow.
[Tables 1 and 2] The formatting of the fps values ('0 .008 𝑓𝑝𝑠') contains stray spaces and uses italic text; the inconsistent spacing should be corrected.
[Section 2 (Methods)] The sentence 'We transform all recorded PV frames to a video prior to frame-wise labeling' is slightly unclear because the video is then processed frame-wise; rephrase to clarify that the frames are assembled into a video for input to SAM-Track.
[Figure 4 caption and Section 5] The caption describes 'distortions in labeling' while the text describes 'incomplete segmentation' of a multi-segment liver; align the terminology and explain in the text what type of distortion occurs (e.g., background leakage vs. missing segments).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: HOLa's evaluation is an external benchmark against independent human annotations, with no fitted parameters or self-referential derivation.

full rationale

The paper contains no derivation chain that reduces to its own inputs. HOLa is a software integration of SAM-Track/DeAOT with a seed-point prompt; the method is fixed before evaluation and no parameters are fitted to the 90 test frames per experiment. The reported Dice scores measure agreement between HOLa's automatically generated masks and a human reference annotation (HA 1), an external ground truth that is produced independently of the algorithm. The human reference was additionally reviewed by two medical experts and cross-compared against four further annotators, which strengthens rather than weakens the external-benchmark character of the evaluation. The acknowledged limitations are threats to external validity, not circularity: the subjective selection of 90 frames 'that best represent the variations during recording' could bias the reported quality upward, and the reference rater being the recorder could inflate concordance, but these are sampling/measurement concerns that do not make the quality claims equivalent to the method's inputs by construction. The 500x speed comparison is likewise based on measured per-frame annotation times for HOLa versus human annotators, not on a fitted quantity being renamed as a prediction. There is no load-bearing self-citation chain, no imported uniqueness theorem, and no ansatz smuggled in via citation. The central claim is therefore self-contained against an external benchmark, so the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted: HOLa uses pretrained SAM and DeAOT as-is, and the only user input (seed point location) is an interaction not a fitted value. The evaluation relies on three domain assumptions about model transferability, ground-truth validity, and frame representativeness, each of which is plausible but not independently verified.

assumptions (3)

domain assumption SAM and DeAOT, pre-trained on large natural image datasets, transfer to HoloLens 2 RGB video of medical scenes without fine-tuning.
The whole method relies on the pretrained models' zero-shot segmentation/tracking capability; the paper evaluates on 5 datasets but does not fine-tune.
domain assumption The reference annotations by HA1, revised by two medical experts, are a valid ground truth for Dice computation.
Dice scores treat HA1 as truth; inter-rater analysis shows human variability, but HA1 may be biased as the recorder.
domain assumption The 90-frame subset selected from each recording is representative of the full sequence.
The paper states the frames 'best represent the variations during recording'; this selection is subjective and could inflate concordance.

how reviews work

0 comments

Cite this review

Pith. "Pith review of HOLa: HoloLens Object Labeling." pith.science (2026). https://pith.science/paper/7O37WP4Q

@misc{pith2026241204945,
  author       = {Pith},
  title        = {Pith review of: HOLa: HoloLens Object Labeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7O37WP4Q}},
  note         = {Machine review of arXiv:2412.04945}
}

read the original abstract

In the context of medical Augmented Reality (AR) applications, object tracking is a key challenge and requires a significant amount of annotation masks. As segmentation foundation models like the Segment Anything Model (SAM) begin to emerge, zero-shot segmentation requires only minimal human participation obtaining high-quality object masks. We introduce a HoloLens-Object-Labeling (HOLa) Unity and Python application based on the SAM-Track algorithm that offers fully automatic single object annotation for HoloLens 2 while requiring minimal human participation. HOLa does not have to be adjusted to a specific image appearance and could thus alleviate AR research in any application field. We evaluate HOLa for different degrees of image complexity in open liver surgery and in medical phantom experiments. Using HOLa for image annotation can increase the labeling speed by more than 500 times while providing Dice scores between 0.875 and 0.982, which are comparable to human annotators. Our code is publicly available at: https://github.com/mschwimmbeck/HOLa

Figures

Figures reproduced from arXiv: 2412.04945 by the authors.

**Figure 1.** The HOLa app consists of two modes. In the recording mode, first, the user points a sphere cursor onto the object of interest by head motion. He selects the cursor position as seed point for segmentation by voice command, which also starts sensor recording. The labeling mode utilizes the SAM [2] seed point prompt to initialize the SAM-Track [3] DeAOT tracker [4] that tracks the segmented object throughout all subs… view at source ↗

**Figure 2.** presents HOLa annotation results for example frames of all experiments [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Annotation deviations between HOLa and human annotators for a sample clinical frame. Human annotations strongly vary at organ boundaries, especially in case of shadows and lack of color contrast between structures [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Applying HOLa on a liver that is divided into multiple segments leads to distortions in labeling. As a consequence, only one segment of the whole liver is labeled. However, HOLa offers to place additional seed points during quality control. man cross-check. Future wor…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

6 extracted references · 4 canonical work pages

[1]

Visualization, registra- tion and tracking techniques for augmented reality guided surgery: a review

Ma L, Huang T, Wang J, Liao H. Visualization, registra- tion and tracking techniques for augmented reality guided surgery: a review. Physics in Medicine & Biology 2022

work page 2022
[2]

Segment anything

Kirillov A, Mintun E, Ravi N, Mao H, Rolland C, Gustafson L, et al. Segment anything. Proceedings of the IEEE/CVF International Conference on Computer Vision 2023

work page 2023
[3]

Segment and track anything

Cheng Y , Li L, Xu Y , Li X, Y ang Z, Wang W, et al. Segment and track anything. arXiv preprint arXiv:2305.06558 2023

arXiv 2023
[4]

Decoupling features in hierarchical propa- gation for video object segmentation

Y ang Z, Y ang Y . Decoupling features in hierarchical propa- gation for video object segmentation. Advances in Neural Information Processing Systems 2022

work page 2022
[5]

Hololens 2 sensor streaming

Dibene JC, Dunn E. Hololens 2 sensor streaming. arXiv preprint arXiv:2211.02648 2022

arXiv 2022
[6]

Segment anything is not always perfect: An investigation of sam on different real- world applications

Ji W, Li J, Bi Q, Liu T, Li W, Cheng L. Segment anything is not always perfect: An investigation of sam on different real- world applications. Machine Intelligence Research 2024

work page 2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Visualization, registra- tion and tracking techniques for augmented reality guided surgery: a review

Ma L, Huang T, Wang J, Liao H. Visualization, registra- tion and tracking techniques for augmented reality guided surgery: a review. Physics in Medicine & Biology 2022

work page 2022

[2] [2]

Segment anything

Kirillov A, Mintun E, Ravi N, Mao H, Rolland C, Gustafson L, et al. Segment anything. Proceedings of the IEEE/CVF International Conference on Computer Vision 2023

work page 2023

[3] [3]

Segment and track anything

Cheng Y , Li L, Xu Y , Li X, Y ang Z, Wang W, et al. Segment and track anything. arXiv preprint arXiv:2305.06558 2023

arXiv 2023

[4] [4]

Decoupling features in hierarchical propa- gation for video object segmentation

Y ang Z, Y ang Y . Decoupling features in hierarchical propa- gation for video object segmentation. Advances in Neural Information Processing Systems 2022

work page 2022

[5] [5]

Hololens 2 sensor streaming

Dibene JC, Dunn E. Hololens 2 sensor streaming. arXiv preprint arXiv:2211.02648 2022

arXiv 2022

[6] [6]

Segment anything is not always perfect: An investigation of sam on different real- world applications

Ji W, Li J, Bi Q, Liu T, Li W, Cheng L. Segment anything is not always perfect: An investigation of sam on different real- world applications. Machine Intelligence Research 2024

work page 2024