REVIEW 5 major objections 5 minor 13 references
The paper claims that raw egocentric screen video alone — repeatedly re-annotated by a vision-language model that alternates between naming the action and inferring the intent — can converge to a compact finite-state model of user behavior
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 12:08 UTC pith:YYKYJGL5
load-bearing objection The pipeline is genuinely new and well-reported, but the evaluation is circular: labels come from the same VLM told to converge, so the predictive claims are unproven until checked against external ground truth. the 5 major comments →
SERUM: State Extraction and Refinement for User Modeling
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
SERUM's central claim is that iterative, context-aware VLM annotation replaces the need for logs, taxonomies, and labels: alternating activity and intent passes reach a stable vocabulary by pass 8, normalization cuts that vocabulary by 46%, and the resulting Markov user models achieve substantially lower perplexity and higher top-1 next-state accuracy than majority, weighted-random, and uniform baselines (normalized activity accuracy 48.5% vs 37.5% raw; normalized intent accuracy 58.2% vs 33.3% raw). The paper further reports that five human raters judge 88.3% of final-pass labels accurate and prefer final-pass labels over first-pass labels 82.8% of the time, and that an ablation removing in
What carries the argument
The alternating multi-pass annotation loop is the load-bearing mechanism: odd passes label observable actions, even passes infer latent intents, and from pass two onward every pass conditions on a run-length-encoded temporal window of neighboring frames plus a compressed narrative summary of all earlier passes. This feedback lets the model revise prior judgments and drives the vocabulary to what the paper calls schematic equilibrium. The second essential component is label normalization: sentence-embedding cosine similarity with a human-calibrated threshold (reported F1 0.82) merges synonymous surface labels, compressing vocabulary by 46% and making the transition statistics of a first-order
Load-bearing premise
The load-bearing premise is that the convergence to a stable label set reflects the video's actual behavioral structure, not the annotator's instructions to collapse synonyms and split intents.
What would settle it
Take a held-out video, randomly permute its frame order, and run the identical multi-pass pipeline. If the label vocabulary still converges by pass 8 and the normalized Markov model still beats frequency baselines on the permuted sequence, then the extracted 'user model' is an artifact of the prompt, not of temporal behavior.
If this is right
- Because no ontology or labels are required, the same pipeline can be pointed at any user's screen recordings; the only domain-specific choices are the merge threshold and the stop pass.
- The pass-8 convergence gives a practical compute bound: running twelve interleaved passes gives a margin without degrading the model.
- Normalization is not a cosmetic fix: cutting vocabulary by about half is what lets Markov transitions be estimated reliably, so downstream systems should budget for synonym merging as part of the model.
- Intent-conditioned labels carry predictive signal that pure activity labels miss; the ablation shows activity-only labels collapse to generic states on coding videos.
- The same harness applied to a large public egocentric cooking corpus shows the gains generalize beyond curated videos, with the method gap over baselines widening after normalization.
Where Pith is reading between the lines
- A natural stress test would be to scramble frame order: if convergence and prediction gains survive, the pipeline is encoding prompt priors rather than temporal structure; if they vanish, the alternating loop is genuinely extracting behavior.
- The framework transfers conceptually to other unstructured behavioral streams — keystroke logs, smart-home sensor traces, browser histories — wherever 'what is the user doing and why' can be asked of a sequence of observations.
- The paper's coding-domain advantage predicts that prediction gains will track how scripted and repetitive an activity is; this could be tested by ranking domains by measured transition entropy and checking that the Markov-minus-baseline accuracy scales with it.
- Reversing the pipeline to generate counterfactual video from perturbed labels would provide an independent check on whether the extracted states are causally meaningful, not just self-consistent.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. SERUM proposes a multi-pass VLM annotation pipeline that takes raw egocentric video and outputs two finite-state user models, one over actions and one over intents. The pipeline alternates activity and intent annotation passes with temporal context windows and global summaries, then merges synonymous labels via Sentence-BERT with a calibrated threshold t*=0.43, and finally fits first-order Markov chains on the normalized label sequences. The paper reports convergence by pass 8 ("schematic equilibrium"), next-state prediction gains over Majority/Weighted Random/Uniform baselines, human-rated label accuracy of 88.3% and preference of 82.8%, together with ablations and an EPIC-KITCHENS-100 generalization study. The central claim is that SERUM is the first framework to extract structured user models from unstructured egocentric video without logs, predefined taxonomies, or labeled data.
Significance. If the claims held, this would be a meaningful step for egocentric user modeling: it would remove the need for event logs and predefined ontologies, and produce interpretable Markov models that could support proactive assistance. Strengths of the paper include publicly released code, demo, and data; a clearly specified pipeline; a 61-video corpus across four domains; and extensive supplementary studies (threshold calibration, pass-by-pass curves, model-scale sensitivity, transferability, and a large EPIC-KITCHENS-100 run). The main weakness is that the current evaluation is largely self-referential: the same VLM produces the labels that are later used as prediction targets, and the temporal context used during annotation incorporates information from the held-out portion of the video. The reported normalized accuracy gaps are also small and highly variable, and the human evaluation is small-scale with low inter-annotator agreement. The paper is therefore better viewed as a conditional contribution whose external validity still needs to be established.
major comments (5)
- [§3.2, §4.3, Appendix G] The central RQ2 evaluation is circular. In §3.2, every pass after Pass 1 conditions on a temporal context window of w=20 neighboring frames and on a full-video summary. Thus the label assigned to a frame in the held-out final 40% already depends on surrounding frames, including future frames within the test portion, and on the same VLM's prior outputs. Evaluating next-state prediction on these labels (§4.3) can therefore reward Markov structure that was injected by the annotation procedure rather than discovered in the video. Appendix G compounds this: the refined prompts explicitly instruct the model to "Collapse synonyms" and "Split actions that serve different intents," actively imposing the compact, intent-split state space that is later presented as an emergent result. To support the headline claim, the authors need a non-circular evaluation: for example, compare SERUM's states agai
- [Table 2, §4.3, §5] The predictive gains over baselines are small relative to the variance and, for raw activity labels, nonexistent. Table 2 shows raw activity Markov accuracy 37.5±33.3 vs. Majority 37.6±34.0; after normalization the gap is only +2.0 pp (48.5 vs. 46.5), and for intent it is +4.8 pp (58.2 vs. 53.4), all with standard deviations around 30 pp. No significance tests or per-video paired comparisons are reported. The conclusion in §5 even acknowledges that several videos achieve near-zero accuracy under split-half evaluation, which is consistent with the hypothesis that the gains are concentrated in a few structured videos. The claim of "substantially" better prediction requires paired effect-size analysis, win-rate statistics, and ideally confidence intervals.
- [§4.4, Ethics Statement] The human evaluation does not establish external label validity. Only 180 frames from 9 videos drawn from a pre-vetted set of 26 are rated, by five uncompensated in-lab volunteers, with Krippendorff's α≈0.40 for both accuracy and preference. The Ethics Statement itself acknowledges that the small in-lab annotator pool may bias results toward positive judgments. Majority-vote agreement with the VLM is not the same as accuracy against ground truth, especially when the annotators see only frames from the same videos and may be influenced by the VLM's language. Independent annotators, a random (not pre-vetted) video sample, and per-item agreement measures are needed; where possible, comparison to dataset ground-truth labels would be stronger.
- [§3.3, §4.1, Appendix F] The normalization threshold t*=0.43 is tuned on a 200-pair calibration set rated by a single annotator and then applied in the same pipeline whose downstream gains are reported in §4.5. Because the threshold is a free parameter selected to maximize F1 on pairs sampled from the same videos and passes used in the evaluation, the reported vocabulary reduction and accuracy improvements may be optimistically biased. A cross-validated or independently collected threshold-calibration set is needed before attributing the +18.2 pp accuracy gain and −14.9 perplexity reduction to normalization per se.
- [Appendix C] The EPIC-KITCHENS-100 generalization study does not validate the extracted states. It applies the same self-label-based Markov harness to 366 additional videos and reports accuracy/perplexity on SERUM-generated labels, but it never compares SERUM's states to the dataset's ground-truth action labels. Since EPIC-KITCHENS-100 provides verb and noun annotations, the paper could directly measure whether SERUM's state vocabulary corresponds to real actions. Without such a comparison, the appendix only demonstrates that the pipeline can be run at larger scale, not that the extracted user models generalize.
minor comments (5)
- [§1] Typo: "platforms like YouTube, and Twitch, . converting raw video" contains a stray comma and period.
- [Abstract and §1] Missing space in "termschematic equilibrium" in multiple places.
- [Table 1] The row "All 61182±94" appears to be a formatting error; it should presumably read "61" and "182±94" as separate columns.
- [§3.3 and §4.1] Sentence-BERT is referred to inconsistently as "SentenceBERT" and "Sentence-BERT"; the internal reference "(§6a)" in §4.1 should be Appendix F.
- [§4.1] The hardware description "two nodes (2 × NVIDIA A5000, 2 × NVIDIA A6000)" is ambiguous about whether each node has two GPUs; please clarify.
Circularity Check
Evaluation is self-referential: the labels used as prediction targets are produced by the same context-conditioned VLM, and the normalization threshold is calibrated on the same label pool, so the Markov gains partly reflect the annotator's self-consistency rather than extracted user behavior.
specific steps
-
fitted input called prediction
[Appendix F (Threshold Calibration Procedure); §4.1 Experimental Setup; Table 2]
"To calibrate the semantic merging threshold, we assemble all unique activity and intent labels across every video and pass, compute pairwise SentenceBERT cosine distances... yielding 100 pairs per type (200 total)... The optimal threshold t∗ is selected as the cosine distance maximizing F1 score on non-skipped pairs..."
The threshold t*=0.43 is fitted on label pairs sampled from all videos and all passes, including the final 40% of frames that are later called 'held-out'. The same threshold is then used to normalize both training and test labels before measuring held-out next-state accuracy. Test labels are therefore transformed by a parameter fitted to those very labels, allowing test-only surface forms to be merged into training states and inflating the apparent Markov prediction gains.
-
self definitional
[§3.2 Multi-Pass Annotation Pipeline; §4.2 RQ1; Appendix G Prompt Templates]
"Passes continue until the label vocabulary stabilizes—empirically by pass 8—a convergence we term schematic equilibrium (§4.2). ... Refined activity ... Collapse synonyms. Split actions that serve different intents."
Schematic equilibrium is defined as the multi-pass annotator reaching a fixed point with respect to its own outputs: each pass conditions on prior-pass labels, and the prompts explicitly instruct the model to collapse synonyms, split actions, and validate or invalidate prior inferences. RQ1's 'stable taxonomy' is therefore convergence of the system to itself, not an independent measurement of user behavior. The claimed phenomenon is partly manufactured by the refinement loop's design.
-
other
[§3.2 Multi-Pass Annotation Pipeline; §3.4 Output: User Models; §4.3 RQ2 Next-State Prediction]
"Each pass (from Pass 2 onward) receives two context signals. First, a temporal context window of w=20 neighboring frames, encoded as a run-length encoding (RLE)... Second, an inter-pass summary—a natural-language summary generated from the full-pass RLE... UMs trained on the first 60% of frames predict the held-out final 40%."
The labels used as evaluation targets are generated by a VLM that sees a 20-frame neighborhood and a full-video summary while annotating each frame. This makes the label sequence autocorrelated and smoothed by construction: the 'held-out' labels are not independent of the training labels or of their own neighbors. A Markov model can exploit this autocorrelation, while the frequency baselines cannot, so the central RQ2 comparison partly measures the VLM's self-consistency rather than any external behavioral structure.
full rationale
The paper does not rely on a self-citation chain or an imported uniqueness theorem, and the Markov-model evaluation is implemented on the authors' own generated labels rather than derived from an equation that equals its input. However, the two main empirical claims are self-referential in an important sense. RQ1's 'schematic equilibrium' is defined as convergence of the iterative VLM to its own prior outputs, with prompts that explicitly demand synonym collapse and self-validation. RQ2's next-state prediction is measured against labels produced by the same context-conditioned VLM, and the normalization threshold used on both train and test labels is calibrated on the full label pool, including the held-out portion. These design choices can manufacture Markovian structure and inflate normalized prediction gains even if the extracted states do not correspond to real user behavior. The human evaluation provides some independent signal, but it is small-scale, in-lab, and exhibits low inter-annotator agreement, so it does not fully break the self-referential loop. This is partial circularity rather than a full reduction of the output to the input by construction.
Axiom & Free-Parameter Ledger
free parameters (4)
- semantic merge threshold t* =
0.43 (cosine distance)
- temporal context window size w =
20 frames
- number of annotation passes =
12 passes (6 activity + 6 intent)
- frame sampling interval =
5 seconds
axioms (5)
- domain assumption VLM-generated labels are treated as valid observations of user behavior.
- domain assumption First-order Markov assumption captures user next-state dynamics.
- domain assumption Sentence-BERT cosine similarity is a valid proxy for semantic synonymy of action and intent labels.
- domain assumption Vocabulary stabilization across passes indicates a meaningful behavioral taxonomy.
- domain assumption Human ratings by five lab colleagues on author pre-vetted videos provide a reliable measure of label quality.
read the original abstract
Agentic assistants capable of proactive, personalized interactions require structured models of user intent and workflow. However, building these models from raw, unstructured screen activity remains an open challenge. We present SERUM, a multi-pass framework that extracts finite-state behavioral models directly from unstructured egocentric video using hierarchical VLM annotation. Processing screen recordings through a sliding window, SERUM alternates between activity-recognition and intent-inference passes, with each pass refining labels using accumulated prior context to reduce hallucination and temporal conflation seen in single-pass annotation. Synonymous states are then merged via sentence embeddings and human-calibrated thresholds into a compact, coherent taxonomy. We evaluate behavioral structure by fitting first-order Markov models over the resulting label sequences (both actions and intents) and measuring predictive accuracy against frequency baselines. Across 61 egocentric videos in four domains (coding, cooking, physical activities, and daily life), we find: (1) iterative label refinement converges to a stable state vocabulary, which we term schematic equilibrium, after several passes; (2) normalized Markov models achieve substantially lower perplexity and higher action predictions than frequency baselines, with the largest gains on structured tasks like coding; and (3) human annotators rate final-pass labels as accurate and meaningfully improved over first-pass labels. To our knowledge, SERUM is the first system to produce interpretable process models from unstructured egocentric screen video without manual annotation, opening a scalable pathway for user modeling and behavioral understanding in the wild. Our demo, code, and results are publicly available
Figures
Reference graph
Works this paper leans on
-
[1]
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966,
-
[7]
Nils Reimers and Iryna Gurevych
arXiv:2207.12080. Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using siamese BERT-networks. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP-IJCNLP), pp. 3982–3992. Association for Computational Linguistics, 2019a. Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings usin...
Pith/arXiv arXiv 2019
-
[8]
Omar Shaikh, Shardul Sapkota, Shan Rizvi, Eric Horvitz, Joon Sung Park, Diyi Yang, and Michael S
arXiv:2508.01943. Omar Shaikh, Shardul Sapkota, Shan Rizvi, Eric Horvitz, Joon Sung Park, Diyi Yang, and Michael S. Bernstein. Creating general user models from computer use,
-
[9]
URL https://arxiv.org/abs/2505.10831. Wil van der Aalst, Arya Adriansyah, Ana Karla Alves de Medeiros, Franco Arcieri, Thomas Baier, Tobias Blickle, Jagadeesh Chandra Bose, Peter van den Brand, Ronald Brandtjen, Joos Buijs, et al. Process mining manifesto. InBusiness Process Management Workshops (BPM 2011), volume 99 ofLecture Notes in Business Informatio...
arXiv 2011
-
[11]
arXiv:2507.17050. A Full Data Collection Table 7: Dataset Overview Video Category Frames Passes Interval Source URL ACS_salestrainingP12 Daily Life 55 12 5syoutu.be/ZG4ExqMVA7w AC_leetcode2P12 Coding 80 12 5syoutu.be/vRAK2YnFr1o AC_leetcodeP12 Coding 352 12 5syoutu.be/zeLZuhi6eYU AC_pizzaP12 Cooking 86 12 5syoutu.be/Q9j6HhF0tGE AC_profreactsP12 Daily Life...
Pith/arXiv arXiv 2026
-
[12]
The intent graphs (d–f) reveal structure invisible in the activity graph. By pass 12, two workflow clusters emerge: an audience-facing loop (responding to viewer comment→speaking into microphone→ creating digital content → preparing tutorial video on intersection observer API), and a production pipeline (managing content schedule → reviewing and refining ...
2026
-
[14]
Table 13: Schematic equilibrium across VLM scales: vocab size by pass
primarily due to larger models being more verbose and specific about label assessments, thereby inflating vocabulary sizes (Table 13). Table 13: Schematic equilibrium across VLM scales: vocab size by pass. 4 Qwen3-VL variants on 4 videos. Intent passes Activity passes Video Model P2 P4 P6 P8 P10 P12 P1 P3 P5 P7 P9 P11 AC_leetcode 4B 18 18 18 18 18 17 5 6 ...
2026
-
[2016]
12 Published as a conference paper at COLM 2026 Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191,
Pith/arXiv arXiv 2026
-
[2020]
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli
doi: 10.1109/TPAMI.2020.2992889. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. ChatGPT outperforms crowd- workers for text-annotation tasks.Proceedings of the National Academy of Sciences, 120 (30):e2305016120,
arXiv 2020
-
[2022]
AnnoLLM: Making large language models to be better crowdsourced annotators
Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. AnnoLLM: Making large language models to be better crowdsourced annotators. InProceedings of the 2024 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics (NAACL),
2024
-
[2023]
doi: 10.1073/pnas.2305016120. arXiv:2303.15056. Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4D: Around the world in 3,000 hours of egocentric video. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p...
-
[2024]
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E
arXiv:2303.16854. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles (SOSP),
-
[2025]
URL https://arxiv.org/abs/2511.21631. Joos C. A. M. Buijs, Boudewijn F. van Dongen, and Wil M. P . van der Aalst. On the role of fitness, precision, generalization and simplicity in process discovery. InOn the Move to 11 Published as a conference paper at COLM 2026 Meaningful Internet Systems: OTM 2012 (CoopIS), volume 7565 ofLecture Notes in Computer Sci...
Pith/arXiv arXiv 2026
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.