Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Casper: Inferring Diverse Intents for Assistive Teleoperation with Vision Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a vision-language-model assistive teleoperation system can infer a user's intent from short teleoperation snippets, execute it via a parameterized skill library, and outperform full teleoperation and closed-set…

desk verdict A solid empirical VLM-assistive-teleoperation paper whose central claim holds, but the unmeasured recall of the candidate generator is the main soft spot; worth a serious referee. read the letter →

arxiv 2506.14727 v2 pith:WU5XNUGV submitted 2025-06-17 cs.RO cs.AI

Huihan Liu , Rutav Shah , Shuijing Liu , Jack Pittenger , Mingyo Seo , Yuchen Cui , Yonatan Bisk , Roberto Martín-Martín
show 1 more author
Yuke Zhu
This is my paper
classification cs.ROcs.AI
keywords assistiveteleoperationsharedautonomyintentinferencevisionlanguagemodelsmobilemanipulationcommonsensereasoninghuman-robotinteractionconfidenceestimation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CASPER is an assistive-teleoperation system built around the idea that a pretrained vision-language model can supply the commonsense a robot needs to guess a human user's goal from a short snippet of teleoperation. The paper claims that by generating an open-world list of plausible skill–object intents from the current scene, selecting among them with VLM reasoning over the user's motion, and only offering help when repeated VLM calls agree, the robot can assist with long-horizon mobile manipulation tasks it was not trained for. In a user study on three multi-step tasks, CASPER reports an average 88.9% task success rate versus 76.4% for full teleoperation and lower success for two assistive baselines, together with lower workload and higher satisfaction. The significance the authors are aiming at is that shared control no longer has to be restricted to predefined goals or task-specific data, because commonsense reasoning can interpret user inputs in the visual context.

What carries the argument

The load-bearing mechanism is two-stage VLM intent inference running in the background while the user teleoperates. A candidate-generation VLM inspects the current scene and proposes feasible skill–target pairs, and a selection VLM chooses the most likely intent from those candidates using a history of subsampled images, robot poses, and visual prompts such as Set-of-Mark labels, a gripper mask, and a motion arrow. A confidence module runs several VLM calls in parallel and only offers help when the agreement count passes a threshold, so ambiguous or incomplete teleoperation is deferred rather than acted on. Confirmed intents are executed by a parameterized skill library spanning navigation, picking, placing, pouring, pushing doors, tapping cards, and pressing buttons, with a separate VLM estimating skill parameters such as target object poses.

What would settle it

An offline experiment could measure candidate recall separately from selection accuracy: after each teleoperation snippet, check whether the user's actual goal appears in the candidate list before selection. If goals are absent from the candidate list in any substantial fraction of trials, that missing recall, not the selector's accuracy, would cap the system's success rate.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that VLM-driven intent inference makes assistive teleoperation work for diverse, open-ended goals. CASPER continuously runs a candidate generator and an intent selector over subsampled images and robot states while the user teleoperates, uses visual prompting to point the VLM at the gripper's motion, and gates any assistance on a self-consistency threshold. The reported evidence is that this outperforms full teleoperation and two prior assistive baselines in task success, completion time, workload, and satisfaction, including cases where commonsense matters, such as pouring pasta into a pot rather than toward a nearby laptop. The paper also reports that the VLM's commonsense reasoning, not just its perception, is the decisive component: removing visual prompting costs accuracy, and confidence estimation lowers false predictions.

Load-bearing premise

The system assumes the model that generates the candidate list always includes the user's real goal among its options; if the true intent is missing from that list, the selector cannot recover it.

Editorial extensions

If this is right

  • Assistive teleoperation can escape closed-set goals, since a pretrained VLM can propose and select intents for objects and scenes the system was never trained on.
  • Long-horizon mobile manipulation becomes reachable because the parameterized skill library composes navigation with contact-rich skills rather than assuming a single predetermined action type.
  • Confidence-gated assistance reduces user disruption, as the robot stays silent until repeated VLM calls agree, and the paper reports lower false prediction rates with this gating.
  • If the user-study results generalize, shared control becomes a more practical option for users who find continuous manual control physically or mentally demanding.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not separate candidate recall from selection accuracy; an untested extension would measure how often the true goal actually appears in the VLM's candidate list, which would pinpoint whether perception or commonsense selection is the binding constraint.
  • Because intents are restricted to skill–target pairs, the framework in principle extends to finer-grained goals such as preferred motion paths or interaction styles, a direction the paper lists as unsupported.
  • The skill library's text-description interface suggests that new skills could be added without retraining the VLM, but no experiment in the paper demonstrates such continual skill acquisition.
  • Performance will likely track the commonsense competence of the chosen VLM backbone, so rerunning the offline intent-inference tests with a different backbone would map how much of the reported benefit is model-specific.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper proposes CASPER, an assistive teleoperation system for mobile manipulators that combines a VLM-based open-world perception module, VLM-powered intent candidate generation and selection from short teleoperation snippets, a self-consistency confidence gate, and a parameterized skill library. The system is evaluated with a TIAGo robot in a user study with 13 participants across three long-horizon tasks (Shelf, Toy, Door), comparing against full teleoperation and two assistive baselines (HAT and RBII), plus offline unit tests and ablations. The authors report higher task success (average 88.9% vs. 76.4%, 40.3%, and 45.0%), lower NASA-TLX workload, and higher user satisfaction for CASPER.

Significance. If the reported results hold, CASPER would be a meaningful advance toward open-world assistive teleoperation, showing that pretrained VLMs can supply commonsense intent inference without task-specific training. The paper has clear strengths: a real-robot user study with long-horizon tasks, a modular architecture, ablations that isolate visual prompting and confidence estimation, and use of pretrained models without finetuning on the evaluation data, so the central claims are not artifacts of fitted parameters. The main weakness is that the system's ceiling is set by the candidate generator's recall, which is never measured independently; this makes the open-world generalization claim the least supported link in the otherwise well-executed empirical study.

major comments (3)
  1. [§3.2, §4.2, Fig. 5] The paper's 'open-world' claim is bounded by the recall of the candidate generator f_candidate, but this quantity is never reported. Section 3.2 splits intent inference into candidate generation and candidate selection; the selector can only choose among the generated candidates, so if the true skill-object pair is absent from the candidate set, no amount of commonsense selection can recover it. Section 4.2 reports only end-to-end intent inference success (correct skill plus correct object), which is the product of candidate recall and conditional selection accuracy; it does not report whether the ground-truth intent was present in the candidate set. Because the three evaluation tasks and the prompt examples in Appendix A.2 are drawn from the same task types, the zero-shot behavior of f_candidate in genuinely novel scenes is not established. I would like to see per-subtask candidate recall, a failure decomposition into candidate misses versus selection errors, and ideally an evaluation on scenes or tasks not reflected in the prompt examples.
  2. [§4.1, Table 1] The statistical support for the headline performance claim is under-reported. The text says CASPER 'exhibits significant improvements (p < 0.05) in task success rate compared to all baselines,' but Table 1 contains no p-values, effect sizes, confidence intervals, or per-participant trial counts. Since success is binary and each participant performed three tasks, it is unclear whether the pairwise t-tests were performed on per-participant success, per-task success, or pooled trials, and whether any correction for multiple comparisons was applied. Please report the analysis unit, the test statistics, and confidence intervals for the success-rate differences, and clarify whether the t-test is appropriate for the outcome distribution.
  3. [§4.1, Independent Variables] The baseline comparison is a hybrid that makes the role of commonsense reasoning hard to isolate. The paper states that 'HAT and RBII only support grasping, so we use CASPER to predict the skill and let the baselines select the target object.' If the skill comes from CASPER, then the baselines inherit CASPER's skill inference, so the comparison does not evaluate the baselines as complete systems. If the skill is provided as an oracle, then the baselines are given information that CASPER must infer. Please state which was the case, report how often CASPER's skill prediction was correct for the user-study subtasks, and discuss how this adaptation affects the interpretation of the comparison. Without this, the claim that the improvement comes from commonsense reasoning rather than from skill execution is not fully supported.
minor comments (5)
  1. [Fig. 5 caption] The phrase 'Casper vs. Capser - No Visual Prompting' contains a typo ('Capser') and should read 'Casper vs. Casper - No Visual Prompting.'
  2. [Abstract, §1, Appendix C] The abstract and Section 1 state that there were 10 pilot study participants, while Appendix C says 'pilot study of group size = 8.' Please reconcile these numbers.
  3. [Table 1] The Shelf row for HAT reads '8 .3' with an extra space; this should be corrected to '8.3.'
  4. [§3.2, §4.1] The values of the VLM agreement threshold η and the number of VLM calls K used in the user study are not reported. Please state these values in the experimental setup, along with the sampling temperature or other decoding parameters used for the parallel GPT-4o calls.
  5. [§4.1, Results] The sentence claiming 'statistically significant improvements (p < 0.05) in all 10 user satisfaction metrics over all baselines' should clarify how the two metrics not applicable to Full Teleop were handled in the comparison.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: CASPER's central claims are empirical user-study and unit-test results, not derived from fitted quantities or self-citation chains.

full rationale

I walked the derivation chain: Section 3.2 defines a two-stage VLM pipeline (fcandidate generates skill-object candidates; fintent selects among them; confidence via self-consistency), and Section 3.3 executes with a parameterized skill library. No equation in the paper fits a parameter to the outcome it later 'predicts.' The success rates in Table 1, intent-inference accuracies in Fig. 5, NASA-TLX scores, and satisfaction scores are all measured end-to-end against direct teleoperation and assistive baselines; GPT-4o is a pretrained model used zero-shot in the user study, and the only tuning choices (confidence threshold eta, history length T=100) are fixed design/ablation choices, not regressed quantities. The few self-citations (e.g., [18] for the skill library and visual prompting) are references to a prior system used as engineering scaffolding, and the evaluation does not depend on that citation's correctness; no uniqueness theorem or prior 'forced choice' argument is imported. The paper's own Limitations section (Sec. 6) and the candidate-stage recall issue in Sec. 3.2 (if the true intent is missing from the generated candidate set, selection cannot recover it) are real generalizability concerns, but they are assumptions about open-set coverage, not circular reductions: the intent prediction is not defined in terms of the evaluation result, and no prediction is equivalent by construction to its input. Thus the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The system introduces no new physical entities or conserved quantities. Its free parameters are the hand-set inference hyperparameters, and its load-bearing assumptions are about how well a pretrained VLM can enumerate and select intents from teleoperation history.

free parameters (3)
  • history length T = 100 timesteps
    Chosen from an offline ablation (Fig. 5 Middle) on the same three tasks, then fixed for the user study; this is a hand-selected hyperparameter.
  • VLM agreement threshold eta = not reported
    Confidence estimation triggers assistance only when K VLM outputs agree beyond a threshold; the paper does not report the numeric value used in the studies.
  • number of VLM calls K = not reported
    Self-consistency runs K parallel VLM calls; K is not specified in the main text or appendix.
assumptions (4)
  • domain assumption User intent can be represented as a pair of a skill and a target object.
    The problem formulation defines z_i = (l_z, o_z) and the Limitations section acknowledges finer-grained intents are not supported.
  • domain assumption The VLM candidate generator produces a candidate set that contains the user's true intent.
    Section 3.2 generates candidates from observations; if the actual goal is omitted, the intent selector cannot choose it.
  • domain assumption Teleoperation snippets convey enough signal for the VLM to disambiguate the correct intent among candidates.
    The intent selection prompt asks the VLM to judge gripper motion and viewpoint changes; this assumes user inputs are informative in the visual context.
  • domain assumption GPT-4o's pretrained commonsense reasoning generalizes to the robot's camera views and tasks without fine-tuning.
    The entire intent inference and candidate generation rely on zero-shot VLM behavior; the paper only evaluates on three tasks in one building.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Casper: Inferring Diverse Intents for Assistive Teleoperation with Vision Language Models." pith.science (2026). https://pith.science/paper/WU5XNUGV

@misc{pith2026250614727,
  author       = {Pith},
  title        = {Pith review of: Casper: Inferring Diverse Intents for Assistive Teleoperation with Vision Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WU5XNUGV}},
  note         = {Machine review of arXiv:2506.14727}
}
read the original abstract

Assistive teleoperation, where control is shared between a human and a robot, enables efficient and intuitive human-robot collaboration in diverse and unstructured environments. A central challenge in real-world assistive teleoperation is for the robot to infer a wide range of human intentions from user control inputs and to assist users with correct actions. Existing methods are either confined to simple, predefined scenarios or restricted to task-specific data distributions at training, limiting their support for real-world assistance. We introduce Casper, an assistive teleoperation system that leverages commonsense knowledge embedded in pre-trained visual language models (VLMs) for real-time intent inference and flexible skill execution. Casper incorporates an open-world perception module for a generalized understanding of novel objects and scenes, a VLM-powered intent inference mechanism that leverages commonsense reasoning to interpret snippets of teleoperated user input, and a skill library that expands the scope of prior assistive teleoperation systems to support diverse, long-horizon mobile manipulation tasks. Extensive empirical evaluation, including human studies and system ablations, demonstrates that Casper improves task performance, reduces human cognitive load, and achieves higher user satisfaction than direct teleoperation and assistive teleoperation baselines. More information is available at https://ut-austin-rpl.github.io/casper/

Figures

Figures reproduced from arXiv: 2506.14727 by the authors.

Figure 1
Figure 1. CASPER infers user intents and offers help when confident. Given user teleoperation input, CASPER uses VLMs to predict human intent using commonsense reasoning. Upon user confirmation, CASPER performs autonomous execution to fulfill the intent using a skill library. CASPER’s background reasoning runs in parallel with foreground human control to minimize disruption. ing pot. Even if the pasta jar is closer to the lap… view at source ↗
Figure 2
Figure 2. CASPER architecture. VLM-based intent inference runs in parallel with human teleoperation. CASPER generates task candidates from observations and infers intent from user inputs among the task can￾didates, repeating until predictions are self-consistent. Once confirmed by the user, CASPER executes the corresponding skill with estimated parameters. 2 Related Work Assistive Teleoperation. Assistive teleoperation offers… view at source ↗
Figure 3
Figure 3. Toy, Shelf, and Door: multi-step mobile manipulation tasks. At each step, the robot disambiguates user intent among multiple plausible goals, selecting the correct one based on user inputs and visual context. confidence, it remains silent, intervening only when it has a confident prediction. This approach allows the user to operate naturally while the system continuously refines its intent inference. 3.3 Fulfilling … view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: User study: user workload and user satisfaction. CASPER consistently outperforms the baselines in terms of user workload (left) and user satisfaction (right) with statistical significance (p < 0.05). Detailed per-task results and full questions of user satisfaction can…
Figure 5
Figure 5. Figure 5: Quantitative results from unit testing and ablation studies. Left: CASPER outperforms all base￾lines in intent inference success rate. Note that no STD is reported for deterministic baselines. The ablation of Casper vs. Capser - No Visual Prompting (VP) highlights the …
Figure 6
Figure 6. Figure 6: Unit testing visualization. To quantitatively validate CASPER’s intent inference accuracy, we conduct unit testing on teleoperation segments collected for each subtask across all three tasks. Each segment serves as an independent data point for evaluating intent infere…
Figure 7
Figure 7. Figure 7: Confidence estimation visualization. CASPER predicts until the intent is clearer, ensuring more accurate assistance. How does confidence estimation mitigate incorrect intent pre￾dictions? We hypothesize that CASPER’s confidence estimation module reduces false predictio…
Figure 8
Figure 8. Figure 8: Per-task results for NASA-TLX and user satisfaction scores. Note that for user satisfaction scores, “assist helpfully” and “correct intent” are not applicable to Full Teleop. Users generally praised CASPER for accuracy and low workload. “... made it easiest to get the …

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. NeuroCommitSSM: Decision-Centric Shared Autonomy for Safe Assistive Manipulation via EEG-EMG-ET Commit Readiness

    cs.RO 2026-07 conditional novelty 6.0 of 10

    A multimodal biosignal model plus a feasibility-gated supervisor cuts false rest-to-move triggers in an assistive robot arm, though the ground-truth rest labels are generated from the very signals it classifies.

  2. Environment Design for Reliable Shared Autonomy with Probabilistic Guarantees

    cs.RO 2026-07 reject novelty 5.0 of 10

    Workspace layout optimization via a linear-Gaussian margin slack improves goal-inference accuracy in simulation, but the claimed 1−α guarantee is not rigorously established and the experimental results contain interna...

Reference graph

Works this paper leans on

85 extracted references · 52 canonical work pages · cited by 2 Pith papers

  1. [1]

    Cantucci and R

    F. Cantucci and R. Falcone. Collaborative autonomy: Human–robot interaction to the test of in- telligent help. Electronics, 11(19), 2022. ISSN 2079-9292. doi:10.3390/electronics11193065. URL https://www.mdpi.com/2079-9292/11/19/3065

    work page doi:10.3390/electronics11193065 2022

  2. [2]

    S. A. Mostafa, M. S. Ahmad, and A. Mustapha. Adjustable autonomy: A systematic literature review. Artificial Intelligence Review, 51, 2019. doi:10.1007/s10462-017-9560-8

    doi:10.1007/s10462-017-9560-8 2019

  3. [3]

    J. Beer, A. Fisk, and W. Rogers. Toward a framework for levels of robot autonomy in human- robot interaction. Journal of Human-Robot Interaction, 3:74, 06 2014. doi:10.5898/JHRI.3.2. Beer

    work page doi:10.5898/jhri.3.2 2014

  4. [4]

    M. A. Collier, R. Narayan, and H. Admoni. The sense of agency in assistive robotics using shared autonomy, 2025. URL https://arxiv.org/abs/2501.07462

    arXiv 2025

  5. [5]

    J. D. Loehr. The sense of agency in joint action: An integrative review. Psychonomic Bulletin & Review, 29(4):1089–1117, 2022

    2022

  6. [6]

    W. Wen, Y . Kuroki, and H. Asama. The sense of agency in driving automation. Frontiers in Psychology, 10:2691, 2019

    2019

  7. [7]

    A. D. Dragan and S. S. Srinivasa. A policy-blending formalism for shared control. The International Journal of Robotics Research , 32:790 – 805, 2013. URL https://api. semanticscholar.org/CorpusID:18131716

    2013

  8. [8]

    A. D. Dragan and S. S. Srinivasa. Formalizing assistive teleoperation. In Robotics: Science and Systems, pages 73–80. Sydney, Australia, 2012

    2012

Show all 85 references
  1. [9]

    B. D. Argall. Autonomy in rehabilitation robotics: An intersection. Annual Review of Control, Robotics, and Autonomous Systems, 1(1):441–463, 2018

    2018

  2. [10]

    S. Chen, J. Gao, S. Reddy, G. Berseth, A. D. Dragan, and S. Levine. Asha: Assistive teleop- eration via human-in-the-loop reinforcement learning, 2022. URL https://arxiv.org/ abs/2202.02465. 9

    2022 arXiv

  3. [11]

    S. W. Brose, D. J. Weber, B. Salatin, G. G. Grindle, H. Wang, J. J. Vazquez, and R. A. Cooper. The role of assistive robotics in the lives of persons with disability. American journal of physical medicine & rehabilitation , 89 6:509–21, 2010. URL https://api. semanticscholar.o...

    2010

  4. [12]

    D. P. Miller. Assistive robotics: An overview. In Assistive Technology and Artificial Intelli- gence, 1998. URL https://api.semanticscholar.org/CorpusID:2877042

    1998

  5. [13]

    Belsare, Z

    A. Belsare, Z. Karimi, C. Mattson, and D. S. Brown. Toward zero-shot user intent recognition in shared autonomy, 2025. URL https://arxiv.org/abs/2501.08389

    2025 arXiv

  6. [14]

    M. Zhao, R. Simmons, H. Admoni, and A. Bajcsy. Conformalized teleoperation: Confidently mapping human inputs to high-dimensional robot actions, 2024. URL https://arxiv. org/abs/2406.07767

    2024 arXiv

  7. [16]

    Y . Cui, S. Karamcheti, R. Palleti, N. Shivakumar, P. Liang, and D. Sadigh. No, to the right: Online language corrections for robotic manipulation via shared autonomy. In Pro- ceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction, HRI ’23. ACM, Mar....

    2023

  8. [17]

    Karamcheti, M

    S. Karamcheti, M. Srivastava, P. Liang, and D. Sadigh. Lila: Language-informed latent ac- tions. In A. Faust, D. Hsu, and G. Neumann, editors, Proceedings of the 5th Conference on Robot Learning , volume 164 of Proceedings of Machine Learning Research , pages 1379–

  9. [18]

    R. Shah, A. Yu, Y . Zhu, Y . Zhu, and R. Mart´ın-Mart´ın. Bumble: Unifying reasoning and acting with vision-language models for building-wide mobile manipulation, 2024. URL https: //arxiv.org/abs/2410.06237

    2024 arXiv

  10. [19]

    X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou. Self- consistency improves chain of thought reasoning in language models, 2023. URL https: //arxiv.org/abs/2203.11171

    2023 arXiv

  11. [20]

    https://pal-robotics.com/robot/tiago

    TIAGo - mobile manipulator robot. https://pal-robotics.com/robot/tiago

  12. [21]

    Padmanabha, J

    A. Padmanabha, J. Yuan, J. Gupta, Z. Karachiwalla, C. Majidi, H. Admoni, and Z. Erickson. V oicepilot: Harnessing llms as speech interfaces for physically assistive robots. InProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology , UIST ’24, pag...

    2024

  13. [22]

    R. K. Jenamani, P. Sundaresan, M. Sakr, T. Bhattacharjee, and D. Sadigh. Flair: Feeding via long-horizon acquisition of realistic dishes, 2024. URL https://arxiv.org/abs/ 2407.07561

    2024 arXiv

  14. [23]

    S. Liu, A. Hasan, K. Hong, R. Wang, P. Chang, Z. Mizrachi, J. Lin, D. L. McPherson, W. A. Rogers, and K. Driggs-Campbell. Dragon: A dialogue-based robot for assistive navigation with visual language grounding. IEEE Robotics and Automation Letters , 9(4):3712–3719, 2024. 10

    2024

  15. [24]

    Karamcheti, A

    S. Karamcheti, A. J. Zhai, D. P. Losey, and D. Sadigh. Learning visually guided latent actions for assistive teleoperation. In A. Jadbabaie, J. Lygeros, G. J. Pappas, P. Parrilo, B. Recht, C. J. Tomlin, and M. N. Zeilinger, editors, Proceedings of the 3rd Conference on Learnin...

    2021

  16. [25]

    You and K

    E. You and K. K. Hauser. Assisted teleoperation strategies for aggressively controlling a robot arm with 2d input. In Robotics: Science and Systems , 2011. URL https://api. semanticscholar.org/CorpusID:17514416

    2011

  17. [26]

    Broad, T

    A. Broad, T. D. Murphey, and B. Argall. Learning models for shared control of human- machine systems with unknown dynamics. ArXiv, abs/1808.08268, 2017. URL https: //api.semanticscholar.org/CorpusID:2559222

    2017 arXiv

  18. [27]

    Javdani, H

    S. Javdani, H. Admoni, S. Pellegrinelli, S. S. Srinivasa, J. A. Bagnell, and S. J. Carnegie. Shared autonomy via hindsight optimization for teleoperation and teaming. The Inter- national Journal of Robotics Research , 37:717 – 742, 2017. URL https://api. semanticscholar.org/Co...

    2017

  19. [28]

    D. E. Gopinath and B. Argall. Active intent disambiguation for shared control robots. IEEE Transactions on Neural Systems and Rehabilitation Engineering , 28:1497–1506, 2020. URL https://api.semanticscholar.org/CorpusID:216029050

    2020

  20. [29]

    Hoffman, T

    G. Hoffman, T. Bhattacharjee, and S. Nikolaidis. Inferring human intent and predicting human action in human–robot collaboration. Annual Review of Control, Robotics, and Autonomous Systems, 7, 2024

    2024

  21. [30]

    Admoni and S

    H. Admoni and S. S. Srinivasa. Predicting user intent through eye gaze for shared auton- omy. In AAAI Fall Symposia , 2016. URL https://api.semanticscholar.org/ CorpusID:53307292

    2016

  22. [31]

    Brooks and D

    C. Brooks and D. Szafir. Balanced information gathering and goal-oriented actions in shared autonomy. In 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 85–94, 2019. doi:10.1109/HRI.2019.8673192

    2019

  23. [32]

    Nikolaidis, Y

    S. Nikolaidis, Y . X. Zhu, D. Hsu, and S. Srinivasa. Human-robot mutual adaptation in shared autonomy. In Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction, HRI ’17. ACM, Mar. 2017. doi:10.1145/2909824.3020252. URL http://dx. doi.org/10.1145/...

    2017

  24. [33]

    B. A. Newman, R. M. Aronson, S. S. Srinivasa, K. Kitani, and H. Admoni. Harmonic: A multimodal dataset of assistive human-robot collaboration, 2020. URL https://arxiv. org/abs/1807.11154

    2020 arXiv

  25. [34]

    Yoneda, L

    T. Yoneda, L. Sun, G. Yang, B. Stadie, and M. Walter. To the noise and back: Diffusion for shared autonomy, 2023. URL https://arxiv.org/abs/2302.12244

    2023 arXiv

  26. [35]

    Jonnavittula, S

    A. Jonnavittula, S. A. Mehta, and D. P. Losey. Learning to share autonomy from re- peated human-robot interaction. ArXiv, abs/2205.09795, 2022. URL https://api. semanticscholar.org/CorpusID:248965455

    2022 arXiv

  27. [36]

    Zurek, A

    M. Zurek, A. Bobu, D. S. Brown, and A. D. Dragan. Situational confidence assistance for lifelong shared autonomy. 2021 IEEE International Conference on Robotics and Automa- tion (ICRA), pages 2783–2789, 2021. URL https://api.semanticscholar.org/ CorpusID:233231464. 11

    2021

  28. [37]

    C. B. Schaff and M. R. Walter. Residual policy learning for shared autonomy. ArXiv, abs/2004.05097, 2020. URL https://api.semanticscholar.org/CorpusID: 215737315

    2004 arXiv

  29. [38]

    B. Lai, S. Toyer, T. Nagarajan, R. Girdhar, S. Zha, J. M. Rehg, K. Kitani, K. Grauman, R. Desai, and M. Liu. Human action anticipation: A survey, 2024. URL https://arxiv.org/ abs/2410.14045

    2024 arXiv

  30. [39]

    E. V . Mascaro, H. Ahn, and D. Lee. Intention-conditioned long-term human egocentric action anticipation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 6048–6057, 2023

    2023

  31. [40]

    Girase, H

    H. Girase, H. Gang, S. Malla, J. Li, A. Kanehara, K. Mangalam, and C. Choi. Loki: Long term and key intentions for trajectory prediction, 2021. URL https://arxiv.org/abs/ 2108.08236

    2021 arXiv

  32. [41]

    Rasouli, I

    A. Rasouli, I. Kotseruba, T. Kunic, and J. Tsotsos. Pie: A large-scale dataset and models for pedestrian intention estimation and trajectory prediction. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 6261–6270, 2019. doi:10.1109/ICCV .2019. 00636

    2019 doi

  33. [42]

    B. Liu, E. Adeli, Z. Cao, K.-H. Lee, A. Shenoi, A. Gaidon, and J. C. Niebles. Spatiotemporal relationship reasoning for pedestrian intent prediction, 2020. URL https://arxiv.org/ abs/2002.08945

    2020 arXiv

  34. [43]

    Huang, J

    Z. Huang, J. Pohovey, A. Yammanuru, and K. Driggs-Campbell. Lit: Large language model driven intention tracking for proactive human-robot collaboration – a robot sous-chef applica- tion, 2024. URL https://arxiv.org/abs/2406.13787

    2024 arXiv

  35. [44]

    H. Ali, P. Allgeuer, and S. Wermter. Comparing apples to oranges: Llm-powered multimodal intention prediction in an object categorization task, 2024. URL https://arxiv.org/ abs/2404.08424

    2024 arXiv

  36. [45]

    Huang, Y .-J

    Z. Huang, Y .-J. Mun, X. Li, Y . Xie, N. Zhong, W. Liang, J. Geng, T. Chen, and K. Driggs- Campbell. Hierarchical intention tracking for robust human-robot collaboration in industrial assembly tasks, 2023. URL https://arxiv.org/abs/2203.09063

    2023 arXiv

  37. [46]

    S. Liu, P. Chang, Z. Huang, N. Chakraborty, K. Hong, W. Liang, D. L. McPherson, J. Geng, and K. Driggs-Campbell. Intention aware robot crowd navigation with attention-based interaction graph. In IEEE International Conference on Robotics and Automation (ICRA), pages 12015– 12021, 2023

    2023

  38. [47]

    C. Wang, C. P ´erez-D’Arpino, D. Xu, L. Fei-Fei, C. K. Liu, and S. Savarese. Co-gail: Learning diverse strategies for human-robot collaboration. In Proceedings of the 5th Conference on Robot Learning (CoRL), pages 1279–1290, 2021

    2021

  39. [48]

    S. Liu, P. Chang, H. Chen, N. Chakraborty, and K. Driggs-Campbell. Learning to navigate intersections with unsupervised driver trait inference. In IEEE International Conference on Robotics and Automation (ICRA), 2022

    2022

  40. [49]

    H. Wang, K. Kedia, J. Ren, R. Abdullah, A. Bhardwaj, A. Chao, K. Y . Chen, N. Chin, P. Dan, X. Fan, G. Gonzalez-Pumariega, A. Kompella, M. A. Pace, Y . Sharma, X. Sun, N. Sunkara, and S. Choudhury. Mosaic: A modular system for assistive and interactive cooking, 2024

    2024

  41. [50]

    Chang, S

    P. Chang, S. Liu, H. Chen, and K. Driggs-Campbell. Robot sound interpretation: Combining sight and sound in learning-based control. InIEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020. 12

    2020

  42. [51]

    J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg, S. Rusinkiewicz, and T. Funkhouser. Tidybot: Personalized robot assistance with large language models. Au- tonomous Robots, 2023

    2023

  43. [52]

    H. Wang, N. Chin, G. Gonzalez-Pumariega, X. Sun, N. Sunkara, M. A. Pace, J. Bohg, and S. Choudhury. APRICOT: Active preference learning and constraint-aware task planning with LLMs. In Conference on Robot Learning, 2024

    2024

  44. [53]

    Bommasani, D

    R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, E. Brynjolfsson, S. Buch, D. Card, R. Castellon, N. S. Chatterji, A. S. Chen, K. A. Creel, J. Davis, D. Demszky, C. Donahue, M. K. B. Doum- bouya, E. Dur...

    2021 arXiv

  45. [54]

    Firoozi, J

    R. Firoozi, J. Tucker, S. Tian, A. Majumdar, J. Sun, W. Liu, Y . Zhu, S. Song, A. Kapoor, K. Hausman, et al. Foundation models in robotics: Applications, challenges, and the future. The International Journal of Robotics Research, 2023

    2023

  46. [55]

    P. Zhi, Z. Zhang, M. Han, Z. Zhang, Z. Li, Z. Jiao, B. Jia, and S. Huang. Closed-loop open- vocabulary mobile manipulation with gpt-4v. ArXiv, abs/2404.10220, 2024. URL https: //api.semanticscholar.org/CorpusID:269157231

    2024 arXiv

  47. [56]

    Liang, W

    J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. R. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. 2023 IEEE Interna- tional Conference on Robotics and Automation (ICRA) , pages 9493–9500, 2022. URL https://api.semanticschola...

    2023

  48. [57]

    Z. Hu, F. Lucchetti, C. Schlesinger, Y . Saxena, A. Freeman, S. Modak, A. Guha, and J. Biswas. Deploying and evaluating llms to program service mobile robots. IEEE Robotics and Au- tomation Letters, 9:2853–2860, 2023. URL https://api.semanticscholar.org/ CorpusID:265294597

    2023

  49. [58]

    M. Ahn, A. Brohan, N. Brown, Y . Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakr- ishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y . Kuang, K.-H. Lee...

    2022

  50. [59]

    Huang, F

    W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. R. Florence, A. Zeng, J. Tompson, I. Mordatch, Y . Chebotar, P. Sermanet, N. Brown, T. Jackson, L. Luu, S. Levine, K. Hausman, and B. Ichter. Inner monologue: Embodied reasoning through planning with language models. ArXiv, abs/...

    2022 arXiv

  51. [60]

    Zhang and H

    B. Zhang and H. Soh. Large language models as zero-shot human models for human-robot in- teraction. 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7961–7968, 2023. URL https://api.semanticscholar.org/CorpusID: 257378614

    2023

  52. [61]

    A. Z. Ren, A. Dixit, A. Bodrova, S. Singh, S. Tu, N. Brown, P. Xu, L. Takayama, F. Xia, J. Varley, Z. Xu, D. Sadigh, A. Zeng, and A. Majumdar. Robots that ask for help: Uncertainty alignment for large language model planners, 2023. URL https://arxiv.org/abs/ 2307.01928

    2023 arXiv

  53. [62]

    Chang, S

    P. Chang, S. Liu, and K. Driggs-Campbell. Learning visual-audio representations for voice- controlled robots. In IEEE International Conference on Robotics and Automation (ICRA) , 2023

    2023

  54. [63]

    Chang, S

    P. Chang, S. Liu, T. Ji, N. Chakraborty, K. Hong, and K. R. Driggs-Campbell. A data-efficient visual-audio representation with intuitive fine-tuning for voice-controlled robots. In Confer- ence on Robot Learning (CoRL), 2023

    2023

  55. [64]

    Bahng, A

    H. Bahng, A. Jahanian, S. Sankaranarayanan, and P. Isola. Exploring visual prompts for adapt- ing large-scale models, 2022. URL https://arxiv.org/abs/2203.17274

    2022 arXiv

  56. [65]

    Nasiriany, F

    S. Nasiriany, F. Xia, W. Yu, T. Xiao, J. Liang, I. Dasgupta, A. Xie, D. Driess, A. Wahid, Z. Xu, Q. Vuong, T. Zhang, T.-W. E. Lee, K.-H. Lee, P. Xu, S. Kirmani, Y . Zhu, A. Zeng, K. Hausman, N. Heess, C. Finn, S. Levine, and B. Ichter. Pivot: Iterative visual prompting elicits...

    2024 arXiv

  57. [66]

    J. Yang, H. Zhang, F. Li, X. Zou, C. Li, and J. Gao. Set-of-mark prompting unleashes extraordi- nary visual grounding in gpt-4v, 2023. URL https://arxiv.org/abs/2310.11441

    2023 arXiv

  58. [67]

    NASA-TLX — Wikipedia, The Free Encyclopedia, 2025

    Wikipedia contributors. NASA-TLX — Wikipedia, The Free Encyclopedia, 2025. URL https://en.wikipedia.org/wiki/NASA-TLX. Accessed: 2025-01-24

    2025

  59. [68]

    Grannen, S

    J. Grannen, S. Karamcheti, S. Mirchandani, P. Liang, and D. Sadigh. V ocal sandbox: Contin- ual learning and adaptation for situated human-robot collaboration. In Conference on Robot Learning (CoRL), 2024

    2024

  60. [69]

    Parakh, A

    M. Parakh, A. Fong, A. Simeonov, T. Chen, A. Gupta, and P. Agrawal. Lifelong robot learning with human assisted language planners. In IEEE International Conference on Robotics and Automation (ICRA), pages 523–529, 2024

    2024

  61. [70]

    A. Z. Ren, A. Dixit, A. Bodrova, S. Singh, S. Tu, N. Brown, P. Xu, L. Takayama, F. Xia, J. Varley, Z. Xu, D. Sadigh, A. Zeng, and A. Majumdar. Robots that ask for help: Uncertainty alignment for large language model planners. InConference on Robot Learning (CoRL), 2023

    2023

  62. [71]

    W. Wan, Y . Zhu, R. Shah, and Y . Zhu. Lotus: Continual imitation learning for robot ma- nipulation through unsupervised skill discovery, 2024. URL https://arxiv.org/abs/ 2311.02058

    2024 arXiv

  63. [72]

    Mahadevan, J

    K. Mahadevan, J. Chien, N. Brown, Z. Xu, C. Parada, F. Xia, A. Zeng, L. Takayama, and D. Sadigh. Generative expressive robot behaviors using large language models. InProceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction, HRI ’24, page 482–491. AC...

    2024

  64. [73]

    T. Ren, S. Liu, A. Zeng, J. Lin, K. Li, H. Cao, J. Chen, X. Huang, Y . Chen, F. Yan, Z. Zeng, H. Zhang, F. Li, J. Yang, H. Li, Q. Jiang, and L. Zhang. Grounded sam: Assembling open- world models for diverse visual tasks, 2024. URL https://arxiv.org/abs/2401. 14159. 14

    2024

  65. [74]

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, J. Zhu, and L. Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection, 2024. URL https://arxiv.org/abs/2303.05499. 15 A Methods A.1 Parametrized Skill Lib...

    2024 arXiv

  66. [76]

    Note: - The robot can only manipulate objects that are within 0.7 meters of the robot

    Prompt for Intent Candidates Generation First, give a list of possible tasks to perform, using the information of the scene, the relevant objects, and relevant skills. Note: - The robot can only manipulate objects that are within 0.7 meters of the robot. If the distance from t...

  67. [77]

    The images are taken from the camera on a mobile robot that is moving its base

    Prompt for Human Intent Selection INSTRUCTIONS: You are given a sequence of images of the scene. The images are taken from the camera on a mobile robot that is moving its base. Your goal is to determine the robot’s intent based on this sequence of robot observations. You want ...

  68. [78]

    Consider the given task choices

  69. [79]

    17 If the robot gripper is moving, see where the gripper (as masked in the image) is moving towards based on the green arrow, and use that to determine the task choice option

    Focus on the images, and see if there is a change in robot’s point of view; see how it is moving and changing its position, or if the gripper is getting closer to one of the objects, or turning towards one of the landmarks. 17 If the robot gripper is moving, see where the grip...

  70. [80]

    Finally, answer: What is the robot trying to do? Choose from the list of possible task choices

    Then, summarize the previous executions made by the robot and feedback received from the human or environment. Finally, answer: What is the robot trying to do? Choose from the list of possible task choices. Example reasoning 1: The robot is moving towards the left, where there...

  71. [81]

    assist helpfully

    CASPER : most preferred, easiest to use, and most reliable 19 mental demandphysical demandtemporal demand performance effort frustration 1 2 3 4 5 6 7Demand (1 7) Shelf NASA-TLX ( ) ease of uselow effortlow workloadlow challenge trust confidence safety would reuseassist helpfu...

  72. [82]

    I need to do all the control work by myself for [Full Teleop]

    Full Teleop: precise but demanding, slow, and effort-intensive 20 Full Teleop was seen as high-effort and slow due to its reliance on full teleoperation, making it the most mentally and physically demanding. Users found it precise but burdensome, requiring them to control ever...

  73. [83]

    was a bit alarming

    HAT: unreliable, sometimes alarming, high failure rate HAT had mixed to negative feedback, with users describing it as unreliable, inconsistent, and prone to errors in both prediction and execution. While some users found its intent inference acceptable, execution failures mad...

  74. [84]

    The inferred intent is always wrong, and the control over the robot actions are always weird and potentially have security issues ... Would it accelerate towards me and hit me?

    RBII: unpredictable, imprecise, and occasionally concerning RBII also received mixed to negative feedback, primarily due to its unreliable intent inference and unpredictable behavior. Users frequently expressed concerns about safety and control transparency: “The inferred inte...

  75. [85]

    Although the robot knew what I was going to do, it sometimes failed in finishing the task and led to a state where it was harder to control the robot

    Execution failures at edge cases Some users noted that while the robot correctly inferred their intent, it occasionally failed to execute the task properly, making it harder to recover control: “Although the robot knew what I was going to do, it sometimes failed in finishing t...

  76. [86]

    It was a bit slow while I was in control

    Slow responsiveness and speed Several users felt that CASPER was too slow, particularly when transitioning between teleoperation and autonomous assistance: “It was a bit slow while I was in control.” “Waiting for the robot to respond and move because it’s slow.” “The inference...

  77. [1390]

    URL https://proceedings.mlr.press/v164/ karamcheti22a.html

    PMLR, 08–11 Nov 2022. URL https://proceedings.mlr.press/v164/ karamcheti22a.html

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.