Pith. sign in

REVIEW 4 major objections 6 minor 53 references

This paper shows that a procedural assistant running entirely on a single edge device can guide furniture assembly step by step, answering text questions in about two seconds and visually grounded ones in about eight, without sending video

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 11:47 UTC pith:L6WYF2YV

load-bearing objection A serious systems paper whose headline reliability claim rests on an oracle simulator; worth refereeing if the authors fix the evaluation loop. the 4 major comments →

arxiv 2607.24770 v1 pith:L6WYF2YV submitted 2026-06-09 cs.AI cs.HC

ProcAgent: An Agentic Framework for Procedural Task Guidance on Edge with Human-in-the-Loop

Azizul Zahid , Subrata Biswas , Bashima Islam , Sai Swaminathan This is my paper
classification cs.AI cs.HC
keywords procedural task guidanceedge AIpropose-and-verifyfinite-state task modelhuman-in-the-loopvision-language modelon-device assistantsequence adherence
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

ProcAgent claims that adaptive, agentic procedural assistance can be delivered entirely on edge hardware, without sacrificing usability. The key architectural move is to decouple cheap continuous perception from expensive visual reasoning: a lightweight proposer always runs, but the vision-language model fires only when a candidate event needs verification or a user query needs grounding. A counter-augmented finite-state task graph enforces legal step order, and when perception and task structure disagree, the system asks the human rather than blocking progress. The paper shows that this propose-and-verify design, plus human-in-the-loop confirmation, lifts sequence adherence from 39.4% to 72.8% while keeping text-query latency near two seconds and visual-query latency near eight. If the claim holds, privacy-sensitive domestic guidance no longer requires the cloud.

Core claim

On the paper's own terms, ProcAgent establishes that reliable procedural guidance does not require continuous high-cost vision-language inference. Instead, a CLIP-based proposer continuously emits candidate events, a counter-augmented finite-state model filters them against legal task structure, and a vision-language verifier confirms only plausible state transitions. Confirmed events that violate guards are escalated to a language agent, which asks the user for confirmation rather than acting autonomously. Running entirely on a single NVIDIA Jetson AGX Orin, the system resolves text-only queries in about 2 seconds and visually grounded queries in about 8 seconds, and user confirmation contr

What carries the argument

The central mechanism is the propose-and-verify loop: a low-latency proposer, a counter-augmented finite-state task model, and an on-demand vision-language verifier. The finite-state task model encodes steps as events with guards over integer counters, collapsing many possible orderings into compact self-loops while enforcing preconditions (for example, the table may be flipped only after all four legs are attached). The verifier is invoked only when a proposer candidate claims a state change or when a user query requires visual grounding, ensuring that expensive vision reasoning is applied selectively. The procedural agent's 'reason-before-perception' routing decides which queries need visi

Load-bearing premise

The sequence-adherence result depends on a rule-based simulator that activates user confirmation only when the system makes a sequencing error; if real users confirm or reject differently than that simulator, the headline adherence gain and the 'without sacrificing usability' claim may not hold.

What would settle it

Run the same ablation with real participants whose confirmations and rejections are recorded turn by turn, and compare sequence adherence against the simulator condition; if the 33-point jump does not reproduce, the load-bearing role of user confirmation is an artifact of the simulation. A secondary falsifier is a task with concurrent sub-steps, which the current single-active-step task model cannot represent.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • On-device deployment alone is sufficient for real-time procedural guidance: text queries resolve in roughly two seconds and visually grounded ones in roughly eight, with no cloud round-trip.
  • User confirmation is not a fallback but a primary state-disambiguation signal; disabling it collapses sequence adherence from 72.8% back to 39.4%.
  • The finite-state task model acts as both a procedural controller and an efficiency mechanism, filtering verifier calls and thereby lowering GPU load, power draw, and thermal stress.
  • Larger language-model backbones improve response quality and tool-use routing accuracy at the cost of latency; the 20B-sized agent best balances the two in this instantiation.
  • The framework generalizes to other procedural tasks because only the knowledge base and FSM configuration are task-specific; the proposer, verifier, and agent runtime logic are reused unchanged.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to replace the rule-based confirmation simulator with real human confirmations and measure whether the 33-point sequence-adherence gain reproduces; if users reject valid interventions or confirm errors, the gain may shrink.
  • The same propose-and-verify pattern could extend to cooking, repair, or clinical procedures where privacy and latency matter, but tasks with concurrent or heavily branching steps would stress the single-active-step assumption.
  • The finding that the cheapest interaction mechanism (user confirmation) yields the largest reliability gain suggests that designing for the human as a reliable sensor may be more cost-effective than adding model capacity.
  • The routing decision — when to invoke vision — appears to be as important as generation quality, implying that smaller models that mis-route queries undermine the whole pipeline even if their raw responses are acceptable.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. ProcAgent is a fully on-device, agentic framework for procedural guidance, instantiated for IKEA LACK furniture assembly on a single Jetson AGX Orin. The system couples a continuously running fine-tuned CLIP proposer, an on-demand Qwen2.5-VL-3B visual verifier, a counter-augmented FSM task graph, a GPT-OSS-20B procedural agent, an offline-built RAG knowledge base, and a human-in-the-loop confirmation dialogue. Evaluation is reported along four axes: perception accuracy, reasoning, task-level performance, and user experience, using 15 IKEA Assembly Dataset videos and a 10-participant user study. The paper reports state accuracy of 83.7%, a Sequence Adherence gain from 39.4% (Configuration F) to 72.8% (full ProcAgent), text-query latency of about 2 s, visually grounded query latency of about 8 s, and broadly positive Likert ratings. The central conclusion is that adaptive, agentic procedural assistance can be delivered entirely on edge hardware without sacrificing usability.

Significance. If the reliability claims held, the paper would make a useful architectural contribution: it demonstrates a real multimodal assistant running entirely on a Jetson AGX Orin, with selective invocation of an expensive VLM behind a cheap proposer and an FSM gate. The strengths are concrete and worth acknowledging: the system actually runs on the target hardware; resource, power, and thermal measurements are reported; the component ablation is systematic; the Wizard-of-Oz query bank is a sensible way to decouple query collection from system behavior; and the limitations section is candid about several threats. However, the main quantitative reliability gain — the 33-point Sequence Adherence improvement — is produced by a rule-based simulator that supplies the correct answer whenever a sequencing error occurs, not by any measured user behavior. In addition, the usability claim is supported only by subjective Likert ratings from a single-condition, no-baseline study. These are load-bearing limitations for the paper's central claims, but they are addressable by additional experiments or by re-scoping the claims.

major comments (4)
  1. [§5.1.3, §5.4, Table 6] The headline Sequence Adherence result is produced by an oracle, not by a model of user behavior. §5.1.3 states that the user-confirmation evaluator is 'a rule-based simulator that activates only when the system makes a sequencing error,' and §5.4 uses it to generate the 33-point jump in Table 6 (39.4% -> 72.8%). This is equivalent to assuming perfect user confirmations for precisely the candidates that need adjudication. Real users may reject valid interventions, confirm incorrectly, or ignore the prompt, and the paper reports no real confirmation data anywhere; the user study in §6 contains only Likert ratings. The §7 caveat that 'larger studies with real users would better capture variation in actual confirmations and mistakes' does not repair the headline: the claimed gain is an upper bound under perfect confirmation. The authors should either run a real confirmation study and recomp
  2. [§6, §8] The user study cannot support the claim that ProcAgent helps users assemble furniture more successfully or that usability is not sacrificed, because it has no control condition and no objective outcome measures. The section opens by asking whether the system 'actually help[s] real users assemble furniture more successfully,' but the methods and results report only an 8-item post-session Likert survey after a single ProcAgent session. There is no manual-only or no-assistant baseline, no task-completion rate, no assembly time, no error count, and no behavioral measure of whether users accepted or rejected interventions. The abstract/conclusion claim that usability is preserved therefore rests entirely on subjective self-reports from 10 convenience-sampled participants. The authors should add objective task metrics and at least one comparison condition, or substantially temper the conclusio
  3. [§5.1.4, §5.4, Table 6] The Sequence Adherence metric is defined with configuration-dependent denominators. §5.1.4 explicitly says the metric is 'not a unit-comparable quantity across rows of Table 6,' yet §5.4 and §8 interpret the 39.4% versus 72.8% difference as a 33-point gain attributable to user confirmation. If the denominators differ across configurations, the difference is not a clean effect size. The paper should either report a common-denominator version of the metric (e.g., over a fixed ground-truth candidate set) or avoid cross-row effect-size language entirely.
  4. [§5.1.5, Table 3, Table 5, Appendix A] The response-quality results are scored by three LLM judges from the same family (GPT-4o, GPT-4.1-Mini, GPT-4.1). Because these judges share training data and alignment procedures, agreement among them does not establish independence, and the cross-backbone comparisons in Table 5 may partly reflect GPT-family bias. §7 acknowledges this limitation, but the main comparisons are still presented without the necessary qualification. In addition, §5.1.5 promises 'per-rubric inter-judge agreement' in Appendix A, but Appendix A contains only the evaluation prompt; the agreement numbers are missing. The authors should add judges from at least one other model family and either supply the agreement data or remove the reference to it.
minor comments (6)
  1. [§2.3] Typo: 'It dooes not provide any framework' should be 'It does not provide any framework.'
  2. [§5.4] Typo: 'the task models's contribution' should be 'the task model's contribution.'
  3. [Figure 6 caption] The caption refers to 'Configurations G and Full Sys,' but no Configuration G is defined; the ablation lists Configurations A–F and ProcAgent. Please fix the label or the caption.
  4. [Table 5] The column header 'Tool Use F1' should be 'Routing F1' to match the metric defined in §5.1.5. Also, Table 5 labels GPT-OSS-20B as '(Ours),' but Table 1 describes it as an off-the-shelf procedural agent; please clarify whether this is a pretrained public model or a model contributed by the authors, and provide a citation or training details.
  5. [Appendix A] The rubric in Appendix A uses the term 'Hallucination' for what the main text calls 'Factual Consistency.' The terminology should be harmonized.
  6. [§5.1.2, §7] The evaluation set excludes videos in which users perform multiple sub-steps concurrently. This is a meaningful scope restriction for the 'exactly one active sub-step at a time' assumption and should be stated more prominently, not only in the future-work sentence.

Circularity Check

2 steps flagged

The headline Sequence Adherence gain is produced by a ground-truth oracle, not measured user confirmation, and the response-quality scores are defined as self-consistency with the same Knowledge Base used to generate the responses.

specific steps
  1. self definitional [§5.1.3 and §5.4; Table 6 (ProcAgent row)]
    "For the user-confirmation evaluation, we developed a rule-based simulator that activates only when the system makes a sequencing error; this reflects the intended runtime role of confirmation as a corrective signal rather than a general-purpose interaction channel. ... Because the IKEA demonstration dataset does not include user-induced sequencing errors, user confirmation is simulated using the rule-based mechanism described in §5.1.3, which activates only when the system makes a sequencing error."

    Sequence Adherence is defined as the fraction of reaching candidates correctly classified as consistent or inconsistent with the legal task structure. The confirmation simulator is triggered exactly when the system makes a sequencing error, i.e., precisely when an illegal candidate would otherwise be committed. Thus in the full ProcAgent row every guard-violating candidate is guaranteed to be flagged, so the 33-point jump (39.4% to 72.8%) is the simulator's ground-truth access, not a measured property of real users. The paper's own limitation section concedes real-user variation in confirmations was not captured.

  2. self definitional [§5.1.5 (Response Quality), Appendix A judge prompt]
    "Factual Consistency Score(0–5) measures whether the agent’s response is consistent with the Knowledge Base and the verified current state ... A response that is factually true in the world but contains claims absent from the Knowledge Base is treated as inconsistent under this metric, since the metric evaluates fidelity to the supplied evidence rather than to ground truth about the world [18]. ... All three judges receive the same evaluation inputs, including the Knowledge Base supplied to the agent at session time."

    The response-quality judges are given the same Knowledge Base that the agent uses for retrieval and generation, and Correctness/Factual Consistency are scored against that KB. The resulting scores therefore measure the system's self-consistency with its own input context rather than external correctness. The comparison against the Single VLM baseline is partly forced because the baseline is not supplied with the KB that defines the metric. This is a secondary self-referential loop: the metric is, by construction, fidelity to the very evidence the system was given.

full rationale

The central quantitative reliability claim reduces to an in-loop oracle: the simulated user confirmation always activates on sequencing errors, so Sequence Adherence is equivalent to the simulator's activation rule. This is a genuine, quotable reduction and warrants a score of 6. The response-quality metrics add a second, less central self-referential loop by judging answers against the same Knowledge Base used to generate them. The paper retains substantial independent content: on-device latency, resource and thermal measurements, proposer/verifier selection, and subjective Likert user ratings do not reduce to the oracle. There is no load-bearing self-citation or uniqueness-imported-from-authors chain; the circularity is in the evaluation construction, not in cited prior work. Section 7's admission that 'larger studies with real users would better capture variation in actual confirmations and mistakes' confirms that the headline adherence result was never tied to real user behavior.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

No new physical or theoretical entities are postulated; 'proposer buffer,' 'Session Context,' and the FSM graph are software constructs, not claims about the world. The free parameters are all hyperparameters or models tuned on the evaluation set itself, which inflates the reported numbers and is a form of selection bias rather than a fitted physical constant.

free parameters (3)
  • Perception-stack hyperparameters (window/stride/check timeout, frame-selection strategy) = W=30, S=15, Check=20s; Multi Frame (50-100% RWS)
    Selected from the full hyperparameter search in Appendix B/Table 8 using the same 15 evaluation videos; no separate held-out validation split is reported.
  • CLIP few-shot shot count = 64 shots
    Table 4 compares CLIP at 8/16/32/64 shots and selects 64 shots on frame accuracy and Macro-F1 over the evaluation set.
  • Agent backbone model = GPT-OSS-20B
    Table 5 selects GPT-OSS-20B over Phi-3-Mini and LLaMA-3.1-8B based on quality, routing F1, and latency on the same evaluation set.
axioms (5)
  • domain assumption The IKEA Assembly Dataset gt_segments labels are accurate ground truth for state and sequence.
    All State Accuracy and Sequence Adherence metrics in §5.1.5 and Table 6 compare against these labels.
  • ad hoc to paper The LACK furniture assembly can be fully represented by the hand-configured FSM with 4 phases, 5 events, and counter guards, with exactly one active sub-step at a time.
    §3.3.1 defines the graph; §5.1.2 explicitly excludes videos with concurrent sub-steps because the task model commits to a single active step.
  • domain assumption User confirmations truthfully report whether the user deviated from the expected procedure.
    The confirmation dialogue in §3.5.2 treats the user as final authority, and the rule-based simulator in §5.1.3 activates only on system sequencing errors, implicitly assuming correct confirmations.
  • domain assumption Qwen2.5-VL-3B and GPT-OSS-20B retain sufficient capability after Q5_K_M quantization to perform verification and interaction on-device.
    §4.3 and Table 1 fix quantized versions without independent accuracy benchmarking against their unquantized counterparts.
  • domain assumption A single fixed camera and the exclusion of concurrent sub-steps are sufficient for real domestic procedural assistance.
    §5.1.2 criterion 2 and §7 list occlusion and concurrent execution as future work; the central claim is scoped to this assumption.

pith-pipeline@v1.3.0-alltime-deepseek · 30171 in / 12957 out tokens · 146570 ms · 2026-08-02T11:47:45.928729+00:00 · methodology

0 comments
read the original abstract

Procedural tasks such as furniture assembly and home repair impose substantial cognitive demands because users must interpret instructions, track task progress, reason about spatial state, and recover from errors while performing physical actions. Prior multimodal assistants have shown promise for procedural guidance, but most rely on cloud inference and fixed always-on perception, making them poorly suited to privacy-sensitive, latency-critical domestic settings. We present ProcAgent, a fully on-device, agentic, vision-based procedural assistant for real-time adaptive guidances on a single NVIDIA Jetson AGX Orin. ProcAgent uses a propose-and-verify architecture that combines low-latency continuous perception, a symbolic task graph, on-demand vision-language verification, and an LLM-based interaction agent. The system continuously proposes user progress, invokes expensive visual reasoning only when ambiguity or likely deviation arises, and supports both reactive question answering and proactive intervention with human-in-the- loop confirmation. We evaluate ProcAgent along four dimensions: perception accuracy, reasoning, task-level performance, and user experience. Despite running entirely on-device, the system maintains responsive interaction, resolving text-only queries in approximately 2 seconds and visually grounded queries in approximately 8 seconds. In a user study with 10 participants completing assembly tasks, ProcAgent receives positive ratings for comprehensibility, actionability, and privacy comfort. These results show that adaptive procedural assistance can be achieved entirely on edge hardware without sacrificing usability.

Figures

Figures reproduced from arXiv: 2607.24770 by Azizul Zahid, Bashima Islam, Sai Swaminathan, Subrata Biswas.

Figure 1
Figure 1. Figure 1: ProcAgent, deployed on home assitant device, supports furniture assembly in both reactive and proactive modes. In the reactive mode (A-C), the user explicitly requests help, asks for the next assembly step, and receives visual verification of leg alignment. In the proactive mode (D-E), ProcAgent monitors the task and intervenes when error occurs to confirm whether the legs are securely attached and whether… view at source ↗
Figure 2
Figure 2. Figure 2: System architecture of ProcAgent, an on-device agentic framework for procedural assistance. The framework combines a knowledge base containing RAG context and runtime task knowledge, a two-stage perception stack for current-state proposal and visual verification, a finite state task model for progress tracking and error detection, and a central procedural agent for local, latency-aware reasoning. The agent… view at source ↗
Figure 3
Figure 3. Figure 3: Flowcharts of ProcAgent’s proactive and reactive modes. In the [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Offline knowledge base construction pipeline. Demonstration videos are processed by a three-stage pipeline (atomic step [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Hardware Performance and Thermal Analysis. Comparative benchmarking of ablation configurations (A–F) and the integrated system (ProcAgent) on the NVIDIA Jetson AGX Orin platform. (A) Power-Efficiency Trade-offs: Scatter plot showing the relationship between GPU load and power consumption. The dashed red boundaries define a high thermal stress region (Power > 40W, Load > 80%), where configurations C and E r… view at source ↗
Figure 7
Figure 7. Figure 7: Likert-scale user study results for ProcAgent (n = 10). After using the system, participants completed an 8-item, 5-point questionnaire. The diverging stacked bar chart shows generally positive ratings across most usability and perception dimensions, with especially strong responses for comprehensibility, actionability, confidence, and privacy. 7 Discussion and Limitations Our findings suggest that ProcAge… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

53 extracted references · 2 canonical work pages

  1. [1]

    Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, K...

    Pith/arXiv arXiv 2022

  2. [2]

    Michael Ahn, Debidatta Dwibedi, Chelsea Finn, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Karol Hausman, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Sean Kirmani, Isabel Leal, Edward Lee, Sergey Levine, Yao Lu, Isabel Leal, Sharath Maddineni, Kanishka Rao, Dorsa Sadigh, Pannag Sanketi, Pierre Sermanet, Quan Vuong, Stefan Welker, Fei Xia, Te...

    arXiv 2024

  3. [3]

    Riku Arakawa, Jill Fain Lehman, and Mayank Goel. 2024. PrISM-Q&A: Step-Aware Voice Assistant on a Smartwatch Enabled by Multimodal Procedure Tracking and Large Language Models. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 8, 4, Article 180 (Nov. 2024), 26 pages. doi:10.1145/3699759

    doi:10.1145/3699759 2024

  4. [4]

    Yizhak Ben-Shabat, Xin Yu, Fatemeh Saleh, Dylan Campbell, Cristian Rodriguez-Opazo, Hongdong Li, and Stephen Gould. 2021. The ikea asm dataset: Understanding people assembling furniture through actions, objects and pose. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 847–859

    2021

  5. [5]

    Marius Bock, Hilde Kuehne, Kristof Van Laerhoven, and Michael Moeller. 2024. Wear: An outdoor sports dataset for wearable and egocentric activity recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 8, 4 (2024), 1–21

    2024

  6. [6]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. 2023. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in neural information processing systems 36 (2023), 49250– 49267

    2023

  7. [7]

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. 2018. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European conference on computer vision (ECCV). 720–736

    2018

  8. [8]

    Paul Dourish. 2004. What we talk about when we talk about context. Personal and ubiquitous computing 8, 1 (2004), 19–30

    2004

  9. [9]

    Abdulrahman K Eesee, Vera Varga, György Eigner, and Tamás Ruppert. 2025. Impact of work instruction difficulty on cognitive load and operational efficiency. Scientific Reports 15, 1 (2025), 11028

    2025

  10. [10]

    EMEET. 2020. EMEET OfficeCore Luna Conference Speakerphone. https://emeet.com/products/speakerphone-luna Accessed 2026-05-02

    2020

  11. [11]

    Xiaojie Gao, Yueming Jin, Yonghao Long, Qi Dou, and Pheng-Ann Heng. 2021. Trans-svnet: Accurate phase recognition from surgical videos via hybrid embedding aggregation transformer. InInternational conference on medical image computing and computer-assisted intervention. Springer, 593–603

    2021

  12. [12]

    Hugo Adrian Hartanto, Irena Krishanti Makrie, Violitta Yesmaya, and Permana Gustav Halfian. 2019. The development of furniture assembly instruction based on augmented-reality. ComTech: Computer, Mathematics and Engineering Applications 10, 2 (2019), 75–81

    2019

  13. [13]

    Kimihiro Hasegawa, Wiradee Imrattanatrai, Masaki Asada, Susan Holm, Yuran Wang, Vincent Zhou, Ken Fukuda, and Teruko Mitamura. 2025. ProMQA-Assembly: Multimodal Procedural QA Dataset on Assembly. arXiv preprint arXiv:2509.02949 (2025)

    Pith/arXiv arXiv 2025

  14. [14]

    Patrick Hegemann, Tim Zechmeister, Markus Grotz, Kevin Hitzler, and Tamim Asfour. 2022. Learning symbolic failure detection for grasping and mobile manipulation tasks. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 4302–4309

    2022

  15. [15]

    Yingdong Hu, Fanqi Lin, Tong Zhang, Li Yi, and Yang Gao. 2023. Look before you leap: Unveiling the power of gpt-4v in robotic vision-language planning. arXiv preprint arXiv:2311.17842 (2023)

    Pith/arXiv arXiv 2023

  16. [16]

    Jin Huang, Yuchao Jin, Le An, and Josh Park. 2025. LiteVLM: A low-latency vision-language model inference pipeline for resource-constrained environments. arXiv preprint arXiv:2506.07416 (2025)

    arXiv 2025

  17. [17]

    Sabrina Kletz, Klaus Schoeffmann, Jenny Benois-Pineau, and Heinrich Husslein. 2019. Identifying surgical instruments in laparoscopy using deep learning instance segmentation. In 2019 International Conference on Content-Based Multimedia Indexing (CBMI). IEEE, 1–6

    2019

  18. [18]

    W Kryściński, B McCann, C Xiong, and R Socher. [n. d.]. Evaluating the factual consistency of abstractive text summarization. arXiv 2019. arXiv preprint arXiv:1910.12840 ([n. d.])

    Pith/arXiv arXiv 2019

  19. [19]

    Colin Lea, Michael D Flynn, Rene Vidal, Austin Reiter, and Gregory D Hager. 2017. Temporal convolutional networks for action segmentation and detection. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 156–165

    2017

  20. [20]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems 36 (2023), 34892–34916

    2023

  21. [21]

    Logitech. 2026. Logitech C920s Pro Full HD Webcam with Privacy Shutter. https://www.logitech.com/en-us/shop/p/c920s-pro-hd-webcam Accessed 2026-05-02

    2026

  22. [22]

    Wei Luo, Deyu Zhang, Yin Tang, Fan Wu, and Yaoxue Zhang. 2025. EdgeOAR: Real-time Online Action Recognition On Edge Devices. IEEE Transactions on Mobile Computing (2025)

    2025

  23. [23]

    Hongying Meng, Michael Freeman, Nick Pears, and Chris Bailey. 2008. Real-time human action recognition on an embedded, reconfigurable video processing architecture. Journal of Real-Time Image Processing 3, 3 (2008), 163–176

    2008

  24. [24]

    NVIDIA. 2021. NVIDIA Jetson AGX Orin Series. Technical Report. NVIDIA. https://www.nvidia.com/content/dam/en-zz/Solutions/gtcf21/jetson- orin/nvidia-jetson-agx-orin-technical-brief.pdf

    2021

  25. [25]

    Rohith Peddi, Shivvrat Arya, Bharath Challa, Likhitha Pallapothula, Akshay Vyas, Bhavya Gouripeddi, Qifan Zhang, Jikai Wang, Vasundhara Komaragiri, Eric Ragan, et al. 2024. Captaincook4d: A dataset for understanding errors in procedural activities. Advances in Neural Information Processing Systems 37 (2024), 135626–135679. 32 Zahid et al

    2024

  26. [26]

    Xue Qin, Simin Luan, John See, Cong Yang, and Zhijun Li. 2026. Harnessing embodied agents: Runtime governance for policy-constrained execution. arXiv preprint arXiv:2604.07833 (2026)

    Pith/arXiv arXiv 2026

  27. [27]

    Rafael Radkowski, Jordan Herrema, and James Oliver. 2015. Augmented reality-based manual assembly support with visual features for different degrees of difficulty. International Journal of Human-Computer Interaction 31, 5 (2015), 337–349

    2015

  28. [28]

    Pragnya Ramjee, Bhuvan Sachdeva, Satvik Golechha, Shreyas Kulkarni, Geeta Fulari, Kaushik Murali, and Mohit Jain. 2025. CataractBot: An LLM-powered Expert-in-the-Loop Chatbot for Cataract Patients. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 9, 2, Article 45 (June 2025), 31 pages. doi:10.1145/3729479

    doi:10.1145/3729479 2025

  29. [29]

    Guido Maria Re, James Oliver, and Monica Bordegoni. 2016. Impact of monitor-based augmented reality for on-site industrial manual operations. Cognition, Technology & Work 18, 2 (2016), 379–392

    2016

  30. [30]

    Gabriel Sarch, Yue Wu, Michael Tarr, and Katerina Fragkiadaki. 2023. Open-ended instructable embodied agents with memory-augmented large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023. 3468–3500

    2023

  31. [31]

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. arXiv:2302.04761 [cs.CL] https://arxiv.org/abs/2302.04761

    Pith/arXiv arXiv 2023

  32. [32]

    Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao. 2022. Assembly101: A large-scale multi-view video dataset for understanding procedural activities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 21096–21106

    2022

  33. [33]

    Justin L Sewell, Christy K Boscardin, John Q Young, Olle Ten Cate, and Patricia S O’Sullivan. 2016. Measuring cognitive load during procedural skills training with colonoscopy as an exemplar. Medical education 50, 6 (2016), 682–692

    2016

  34. [34]

    Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. 2020. Finegym: A hierarchical video dataset for fine-grained action understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2616–2625

    2020

  35. [35]

    Zheyu Shen, Yexiao He, Ziyao Wang, Yuning Zhang, Guoheng Sun, Wanghao Ye, and Ang Li. 2025. EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices. InProceedings of the 23rd Annual International Conference on Mobile Systems, Applications and Services (Hilton Anaheim, Anaheim, CA, USA) (MobiSys ’25). Association for Computing Machinery, New...

    arXiv 2025

  36. [36]

    Chunyi Sun, Junlin Han, Weijian Deng, Xinlong Wang, Zishan Qin, and Stephen Gould. 2025. 3d-gpt: Procedural 3d modeling with large language models. In 2025 International Conference on 3D Vision (3DV). IEEE, 1253–1263

    2025

  37. [37]

    Curran, Florian Schaub, and Kang G

    Brian Jay Tang, Kaiwen Sun, Noah T. Curran, Florian Schaub, and Kang G. Shin. 2025. Ads that Talk Back: Implications and Perceptions of Injecting Personalized Advertising into LLM Chatbots. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 9, 4, Article 213 (Dec. 2025), 47 pages. doi:10.1145/3770640

    work page doi:10.1145/3770640 2025

  38. [38]

    Shreshth Tuli, Giuliano Casale, and Manuel Roveri. 2025. SELA: Smart Edge LLM Agent to Optimize Response Trade-offs of AI Assistants. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 9, 3, Article 130 (Sept. 2025), 18 pages. doi:10.1145/3749483

    work page doi:10.1145/3749483 2025

  39. [39]

    Ruiqi Wang, Peiqi Gao, Patrick Lynch, Tingjun Liu, Yejin Lee, Carolyn Baum, Lisa Tabor Connor, and Chenyang Lu. 2025. CHEF-VL: Detecting Cognitive Sequencing Errors in Cooking with Vision-language Models. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 9, 4, Article 220 (Dec. 2025), 35 pages. doi:10.1145/3770714

    doi:10.1145/3770714 2025

  40. [40]

    Xin Wang, Taein Kwon, Mahdi Rad, Bowen Pan, Ishani Chakraborty, Sean Andrist, Dan Bohus, Ashley Feniello, Bugra Tekin, Felipe Vieira Frujeri, et al. 2023. Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 20270–20281

    2023

  41. [41]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629 [cs.CL] https://arxiv.org/abs/2210.03629

    Pith/arXiv arXiv 2023

  42. [42]

    Yaolun Zhang, Xiaogeng Liu, and Chaowei Xiao. 2025. Metaagent: Automatically constructing multi-agent systems based on finite state machines. arXiv preprint arXiv:2507.22606 (2025)

    Pith/arXiv arXiv 2025

  43. [43]

    Ada Yi Zhao, Aditya Gunturu, Ellen Yi-Luen Do, and Ryo Suzuki. 2025. Guided reality: Generating visually-enriched ar task guidance with llms and vision models. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology. 1–15. ProcAgent: An Agentic Framework for Procedural Task Guidance on Edge with Human-in-the-Loop 33 A Mul...

    2025

  44. [44]

    A knowledge base (KB) describing the assembly process

  45. [45]

    The current assembly state

  46. [46]

    The assistant‘s answer INPUTS Knowledge Base (KB): {knowledge_base} Ground Truth Sequence: {ground_truth_sequence} Task Sequence (may be incomplete or user-facing version): {task_sequence} Rules for Completion: {rules_for_completion} Current Assembly State: {current_state} User Question: {question} Assistant Answer: {answer} EV ALUATION CRITERIA Evaluate ...

  47. [47]

    Hallucination (0–5) Does the answer introduce information that is NOT supported by the KB or current state? 5→No hallucination (fully grounded in KB/state) 4→Minor unsupported assumptions, but mostly grounded 3→Some unsupported or speculative content 2→Significant hallucination affecting reliability 1→Mostly hallucinated 0→Completely fabricated / unrelated

  48. [48]

    Correctness (0–5) Is the answer factually correct and consistent with the KB and current state? 5→Fully correct and consistent 4→Minor inaccuracies but overall correct 3→Partially correct with notable mistakes 2→Mostly incorrect 1→Largely incorrect 0→Completely wrong

  49. [49]

    current assembly stage

    Helpfulness (0–5) Does the answer effectively help the user progress in assembly? 5→Clear, actionable, and directly helps progress 4→Helpful but slightly incomplete or unclear 3→Somewhat helpful but lacks clarity or steps 2→Minimally helpful 1→Confusing or not actionable 0→Not helpful at all IMPORTANT INSTRUCTIONS * Ground your evaluation strictly in the ...

  50. [50]

    ONLY use the following standardized action names (Verb-Noun pairs): - Verbs: [pick up, align, spin, attach, flip, insert, tighten, remove, align and spin] - Nouns: [leg, shelf, table, screw, base, panel, drawer, frame]

  51. [51]

    Do NOT summarize

    Cardinality Rule: Log EVERY instance of a repetitive action. Do NOT summarize

  52. [52]

    If an action does not fit the list, use the label "others"

  53. [53]

    reference_video_id

    Follow the Step Extraction Protocol: split complex movements into atomic steps. OUTPUT FORMAT (JSON) Return ONLY the JSON. { "reference_video_id": "{{job_id}}", "furniture_name": "... ", "metadata": { "fps": 30, "video_duration_seconds": 180 }, "steps": [ { "name": "verb + noun", "timestamp": "MM:SS", "frame_index": 0000, "description": "Short visual anch...