Pith. sign in

REVIEW 2 major objections 6 minor 67 references

ImageInThat: Manipulating Images to Convey User Instructions to Robots

T0 review · 2 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that directly dragging objects in a camera image lets users create robot instructions 64.8 percent faster than typing step-by-step text, with higher confidence and no more errors.

desk verdict A well-built system and an honest write-up, but the headline speed claim rests on a self-admittedly unfair text baseline and a t-test that ignores repeated measures. read the letter →

arxiv 2503.15500 v1 pith:4NBYRBFR submitted 2025-01-21 cs.HC cs.RO

Karthik Mahadevan , Blaine Lewis , Jiannan Li , Bilge Mutlu , Anthony Tang , Tovi Grossman This is my paper · ORCID
classification cs.HCcs.RO
keywords directmanipulationrobotinstructionend-userprogrammingimage-basedinterfacestimelineinterfacehuman-robotinteractionkitchentaskslanguage-based
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ImageInThat proposes that people can instruct robots more effectively by directly dragging and clicking objects in a camera image of the robot's environment than by typing step-by-step text commands. In a within-subjects study with ten participants performing four kitchen manipulation tasks, the image-based interface took 110.8 seconds on average versus 363.9 seconds for the text-based method, a 64.8 percent reduction. Participants also reported higher confidence that the robot understood their instructions, rated usability higher, reported lower workload, and preferred the image method. The paper argues that images are intrinsically grounded in the robot's environment, so direct manipulation removes the ambiguity of referring to specific objects and placements in language. It also demonstrates that the resulting image instructions can be translated into robot policy code and executed on a physical arm.

What carries the argument

The carrying mechanism is a timeline-style editor over a segmented visual representation of the robot's environment. Each object is a draggable mask overlaid on background images that encode fixture states (e.g., cabinet open or closed); every user manipulation appends a thumbnail step to the timeline, and the system highlights what changed between consecutive steps to keep the state evolution legible. On top of this, an LLM generates captions for each manipulation, accepts text edits that modify the image, and proposes goal locations and plausible next steps. The same representation is what makes the paradigm work as an instruction format: the image itself is the concrete, grounded specification, and the supporting models translate it into captions or policy code for execution.

What would settle it

Run the same four kitchen tasks with a speech-based or LLM-decomposed text condition; if completion times become statistically indistinguishable from ImageInThat, the central faster-than-text claim no longer generalizes to natural language instruction.

Watch

Extended reading notes

Core claim

The paper's central claim is that direct manipulation of images is a viable and preferable paradigm for conveying step-by-step instructions to a robot. ImageInThat instantiates this by letting users select objects and fixtures in a photographic view and drag them to new locations or toggle their states, with each change automatically inserted as a step in a visual timeline. The system couples these manipulations with automatic captions, language-based image editing, and goal prediction, so the user always sees the environment state that the robot would achieve. The user study found the image condition was 64.8 percent faster than the text condition, with significantly higher confidence, usability, and preference, and no overall error increase; one notable difference was that the text condition produced significantly more missing steps. The paper frames image-based instructions as a middle ground between immediate language commands and traditional end-user programming, supporting both near-real-time commanding and longer-horizon task specification.

Load-bearing premise

The comparison treats typed, step-by-step text as the stand-in for natural-language robot instruction; if a more natural language interface (speech or LLM decomposition) is used, the speed advantage may shrink or vanish.

Editorial extensions

If this is right

  • People can specify a multi-step kitchen task in about a third of the time it takes to type the equivalent step list, with no loss of correctness.
  • Users trust image-based instructions more: they rated confidence and usability higher and reported lower workload than with text.
  • The timeline representation supports the four components of robot instruction: capturing intent, translating it, presenting it for confirmation, and executing it.
  • Image instructions produced with the interface can be turned into executable policy code for a physical robot arm, at least for tasks involving clear object identities.
  • Because missing steps were significantly rarer with images, the format may particularly help in longer-horizon tasks where users must keep track of object states.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported speed advantage is measured against typed text only; the paper itself grants that speech-based or LLM-decomposed natural language could be a stronger baseline, so the headline 'images beat language' should be read as 'images beat typed low-level steps' until such baselines are tested.
  • Image instructions may serve as a shared grounding format across robot policies: goal-image-conditioned policies and vision-language-action models could consume user-edited images directly, bypassing error-prone language grounding.
  • The interaction trace (which objects moved, when, and to where) is a natural dataset for learning user preferences, such as always placing heavy dishes on lower shelves, something the paper mentions only as future work.
  • A direct comparison to speech or to demonstration-based programming would clarify whether the benefit comes from the image modality itself or from the absence of typing.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper introduces ImageInThat, a prototype that lets users instruct a robot by directly manipulating images of the robot's environment in a timeline-style editor, supported by LLM-based captioning, language-to-image editing, and step prediction. The authors report a within-subjects user study (N=10) comparing ImageInThat to a text-based instruction interface across four simulated kitchen tasks, finding a 64.8% reduction in task completion time, higher confidence, higher SUS scores, and no significant overall error difference. The paper also presents a case study translating user-generated image instructions into robot policy code and executing them on a physical robot arm.

Significance. If the empirical findings are valid, ImageInThat offers a compelling alternative paradigm for end-user robot instruction, with concrete grounding in images and potential to reduce ambiguity relative to typed text. The work is significant for HRI and end-user programming: it combines direct manipulation, timeline feedback, and foundation models in a functional prototype, and it includes transparent limitations, bootstrapped confidence intervals, an oracle-based error metric, and released code and supplementary material. However, the central speed claim rests on a task-level paired t-test that violates independence given the nested design, so a reanalysis is necessary before the headline result can be accepted. The comparison baseline is also self-admittedly a worst case for text, so the paradigm-level claim should be scoped accordingly.

major comments (2)
  1. [V] The headline speed comparison is reported as t(37) = −8.96, p <0.001 on 38 task-level observations (10 participants × 4 tasks, minus two corrupted). Because each participant contributes up to four observations and each task is reused across participants, the observations are not independent; a paired t-test on task-level data does not account for the repeated-measures structure, contrary to the claim in Section V that paired t-tests were used 'to account for repeated measures.' Please reanalyze the completion-time, error, and confidence results either as paired t-tests on per-participant means (N=10) or with a mixed-effects model including random intercepts for participant and task, and report the resulting effect sizes, confidence intervals, and p-values. If the speed effect does not survive this reanalysis, the abstract's '64.8% less time' claim is not supported by the current analysis.
  2. [VII] The authors acknowledge in Section VII that the text condition required step-by-step typed instructions, making it a 'worst-case scenario for the text-based method' while ImageInThat is a 'best-case scenario.' Because the abstract and conclusion state the result as a general comparison ('participants were faster with ImageInThat'), the current study does not establish that image manipulation is faster than natural-language instruction generally. Please either add a more representative language condition (e.g., speech input, or an LLM-based decomposition of high-level goals) or consistently qualify every claim, including the abstract, to refer to the specific text-based interface tested. Without this, the paradigm-level conclusion overreaches the evidence.
minor comments (6)
  1. [V] The two sentences reporting extraneous and inefficient steps both give t(37) = 1.35 and report the same means (M = 0.29 vs. M = 0.26), but the condition order is reversed; please correct the typo and define the sign convention for the paired differences so readers can verify the direction.
  2. [IV] Section IV describes the number of proposed next-step actions as 'a system parameter' but does not state the value used in the study; please report the default and any sensitivity to it.
  3. [V] The oracle-based error coding was performed by two experimenters, but no inter-rater reliability measure is reported; please report agreement (e.g., Cohen's kappa) on a subset of tasks.
  4. [V] The manuscript reports several p-values only as inequalities (p < 0.05, p > 0.05) and does not state whether tests are one- or two-tailed; please report exact p-values and test direction.
  5. [VI] In Section VI, 'the robot must be able to execute them, This could be accomplished' has a comma splice; also, 'we illustrate a case study' would be more naturally phrased as 'we present a case study.'
  6. [V] The text callout 'Figure 6 provides a breakdown of participants' completion time' refers only to the middle panel, but the figure contains three panels (errors, time, TLX); please label the panels and reference them specifically in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central claims are direct empirical measurements from a user study, not derivations from fitted inputs or self-cited theorems.

full rationale

This is an empirical systems paper. The central claim — that participants were faster with ImageInThat than with the text-based method — is supported by direct timing measurements in a within-subjects user study (Section V: 'Participants were faster (t(37) = −8.96, p <0.001) with ImageInThat (M = 110.8 seconds, SD = 70.04 seconds) compared to the text-based method (M = 363.88 seconds, SD = 186.45 seconds).'). The authors explicitly acknowledge in Section VII that the comparison represents a best-case scenario for the image paradigm and a worst-case scenario for the text method, which is a limitation of external validity rather than a circular derivation. No parameter is fitted to the outcome, no prediction is generated from a model trained on the same data, and no load-bearing claim rests on a self-citation or on a uniqueness theorem from the authors' prior work. The self-citations present (e.g., [20], [32]) are background references and do not carry the empirical conclusion. The only substantive concern is the repeated-measures analysis treating task-level observations as independent, which is an internal-validity issue, not a circular-derivation issue. Under the hard rules, circularity must be exhibited as Eq. X = Eq. Y by construction or as a fitted parameter renamed as a prediction; neither is present here.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

ImageInThat is an empirical HCI systems paper, so there are no fitted constants in a derivation. The load-bearing assumptions are about the validity of the user study (baseline representativeness, oracle ground truth, statistical independence) and about the feasibility of the environment representation (objects, fixtures, predefined states). The only hand-set numeric parameter is a UI autocomplete count.

free parameters (1)
  • number of plausible next-step actions proposed by the LLM = not specified, described as a system parameter
    Section IV, Predicting user goals: 'generating any number of plausible actions (a system parameter)'. This is a hand-chosen UI parameter that affects the autocomplete feature, not the headline speed or confidence result.
assumptions (4)
  • domain assumption ImageInThat's setup phase can obtain an accurate internal representation of the environment, including objects, fixtures, and fixture states.
    Sections III and IV: Sim uses predefined object/fixture lists and predefined fixture states; Real prompts an LLM for objects but fixture states are predefined. The whole interface depends on this representation.
  • domain assumption The oracle representation created by the two experimenters is a valid ground truth for counting errors.
    Section V Measures: errors are defined relative to an a priori oracle, but no inter-rater reliability or validation of the oracle is reported.
  • domain assumption Task-level observations are statistically independent, justifying paired t-tests on pooled task data.
    Section V Hypotheses: 'we used a paired t-test for all metrics' with t(37), pooling 38 task observations from 10 participants; repeated measures within participants are not modeled, risking pseudo-replication.
  • domain assumption The text-based baseline is a representative instantiation of natural-language robot instruction.
    Section VII Limitations: the authors admit the comparison is best-case for images and worst-case for text, because text required step-by-step typed instructions without speech or LLM decomposition. This assumption is load-bearing for the generalizability of the speed claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ImageInThat: Manipulating Images to Convey User Instructions to Robots." pith.science (2026). https://pith.science/paper/4NBYRBFR

@misc{pith2026250315500,
  author       = {Pith},
  title        = {Pith review of: ImageInThat: Manipulating Images to Convey User Instructions to Robots},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4NBYRBFR}},
  note         = {Machine review of arXiv:2503.15500}
}
read the original abstract

Foundation models are rapidly improving the capability of robots in performing everyday tasks autonomously such as meal preparation, yet robots will still need to be instructed by humans due to model performance, the difficulty of capturing user preferences, and the need for user agency. Robots can be instructed using various methods-natural language conveys immediate instructions but can be abstract or ambiguous, whereas end-user programming supports longer horizon tasks but interfaces face difficulties in capturing user intent. In this work, we propose using direct manipulation of images as an alternative paradigm to instruct robots, and introduce a specific instantiation called ImageInThat which allows users to perform direct manipulation on images in a timeline-style interface to generate robot instructions. Through a user study, we demonstrate the efficacy of ImageInThat to instruct robots in kitchen manipulation tasks, comparing it to a text-based natural language instruction method. The results show that participants were faster with ImageInThat and preferred to use it over the text-based method. Supplementary material including code can be found at: https://image-in-that.github.io/.

Figures

Figures reproduced from arXiv: 2503.15500 by the authors.

Figure 1
Figure 1. We introduce a new paradigm for instructing robots through the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. ImageInThat’s user interface, consisting of an editor (top) and a timeline (bottom). The editor (A) allows users to manipulate objects and fixtures in the environment, while the timeline displays the current state of the environment and the desired changes. The timeline (B) shows all instructions provided to the robot. Selecting a step populates it in the editor. Changes between steps are made visible by contrasting… view at source ↗
Figure 3
Figure 3. System diagram of ImageInThat showing its major components. The server side (in purple) handles the preprocessing step and all intelligent features that require interfacing with the LLM (e.g., autocomplete, captioning, and language-to-step generation). The client is a web user interface built with ReactJS. Components in white are implemented differently for the user study and real-world usage. In Sim, the background… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Sampled results of providing a language instruction to a fine-tuned [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Tasks performed by participants in the evaluation (left to right): [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Left: a plot showing the number of errors for [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 49 canonical work pages

  1. [1]

    Interactive language: Talking to robots in real time,

    C. Lynch, A. Wahid, J. Tompson, T. Ding, J. Betker, R. Baruch, T. Armstrong, and P. Florence, “Interactive language: Talking to robots in real time,” IEEE Robotics and Automation Letters , 2023

    2023

  2. [2]

    A survey of robot learning from demonstration,

    B. D. Argall, S. Chernova, M. Veloso, and B. Browning, “A survey of robot learning from demonstration,” Robotics and autonomous systems, vol. 57, no. 5, pp. 469–483, 2009

    2009

  3. [3]

    A survey on end-user robot programming,

    G. Ajaykumar, M. Steele, and C.-M. Huang, “A survey on end-user robot programming,” ACM Computing Surveys (CSUR) , vol. 54, no. 8, pp. 1–36, 2021

    work page 2021

  4. [4]

    Direct manipulation: A step beyond programming languages,

    B. Shneiderman, “Direct manipulation: A step beyond programming languages,” Computer, vol. 16, no. 08, pp. 57–69, 1983

    work page 1983

  5. [5]

    Directgpt: A direct manipulation interface to interact with large language models,

    D. Masson, S. Malacria, G. Casiez, and D. V ogel, “Directgpt: A direct manipulation interface to interact with large language models,” in Pro- ceedings of the CHI Conference on Human Factors in Computing Systems, 2024, pp. 1–16

    work page 2024

  6. [6]

    Human–robot interaction: a survey,

    M. A. Goodrich, A. C. Schultz et al. , “Human–robot interaction: a survey,”Foundations and Trends® in Human–Computer Interaction, vol. 1, no. 3, pp. 203–275, 2008

    work page 2008

  7. [7]

    Human-in-the-loop optimization of shared autonomy in assistive robotics,

    D. Gopinath, S. Jain, and B. D. Argall, “Human-in-the-loop optimization of shared autonomy in assistive robotics,” IEEE robotics and automa- tion letters, vol. 2, no. 1, pp. 247–254, 2016

    work page 2016

  8. [8]

    Third-person piloting: Increasing situational awareness using a spatially coupled second drone,

    R. Temma, K. Takashima, K. Fujita, K. Sueda, and Y . Kitamura, “Third-person piloting: Increasing situational awareness using a spatially coupled second drone,” in Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology , 2019, pp. 507–519

    work page 2019

Show all 67 references
  1. [9]

    Teleoperation of humanoid robots: A survey,

    K. Darvish, L. Penco, J. Ramos, R. Cisneros, J. Pratt, E. Yoshida, S. Ivaldi, and D. Pucci, “Teleoperation of humanoid robots: A survey,” IEEE Transactions on Robotics , vol. 39, no. 3, pp. 1706–1727, 2023

    2023

  2. [10]

    Still not solved: A call for renewed focus on user-centered teleoperation interfaces,

    D. J. Rea and S. H. Seo, “Still not solved: A call for renewed focus on user-centered teleoperation interfaces,” Frontiers in Robotics and AI , vol. 9, p. 704225, 2022

    2022

  3. [11]

    Effects of onset latency and robot speed delays on mimicry-control teleoperation,

    D. Rakita, B. Mutlu, and M. Gleicher, “Effects of onset latency and robot speed delays on mimicry-control teleoperation,” in HRI’20: Proceed- ings of the 2020 ACM/IEEE International Conference on Human- Robot Interaction , 2020

    2020

  4. [12]

    Multi-view merging for robot teleop- eration with virtual reality,

    D. Wei, B. Huang, and Q. Li, “Multi-view merging for robot teleop- eration with virtual reality,” IEEE Robotics and Automation Letters , vol. 6, no. 4, pp. 8537–8544, 2021

    2021

  5. [13]

    Towards a virtual reality interface for remote robotic teleoperation,

    A. Naceri, D. Mazzanti, J. Bimbo, D. Prattichizzo, D. G. Caldwell, L. S. Mattos, and N. Deshpande, “Towards a virtual reality interface for remote robotic teleoperation,” in 2019 19th International Conference on Advanced Robotics (ICAR) . IEEE, 2019, pp. 284–289

    2019

  6. [14]

    An autonomous dynamic camera method for effective remote teleoperation,

    D. Rakita, B. Mutlu, and M. Gleicher, “An autonomous dynamic camera method for effective remote teleoperation,” in Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction, 2018, pp. 325–333

    2018

  7. [15]

    A comparison of remote robot teleoperation interfaces for general object manipulation,

    D. Kent, C. Saldanha, and S. Chernova, “A comparison of remote robot teleoperation interfaces for general object manipulation,” in Proceedings of the 2017 ACM/IEEE international conference on human-robot interaction, 2017, pp. 371–379

    2017

  8. [16]

    Starhopper: A touch interface for remote object-centric drone navigation,

    J. Li, R. Balakrishnan, and T. Grossman, “Starhopper: A touch interface for remote object-centric drone navigation,” in Proceedings of the Graphics Interface Conference 2020 , 2020

    2020

  9. [17]

    Communicating robot arm motion intent through mixed reality head-mounted displays,

    E. Rosen, D. Whitney, E. Phillips, G. Chien, J. Tompkin, G. Konidaris, and S. Tellex, “Communicating robot arm motion intent through mixed reality head-mounted displays,” in Robotics research: The 18th inter- national symposium ISRR . Springer, 2020, pp. 301–316

    2020

  10. [18]

    Robot teleoperation with augmented reality virtual surrogates,

    M. E. Walker, H. Hedayati, and D. Szafir, “Robot teleoperation with augmented reality virtual surrogates,” in 2019 14th ACM/IEEE Inter- national Conference on Human-Robot Interaction (HRI) . IEEE, 2019, pp. 202–210

    2019

  11. [19]

    Robot programming through augmented trajectories in augmented reality,

    C. P. Quintero, S. Li, M. K. Pan, W. P. Chan, H. M. Van der Loos, and E. Croft, “Robot programming through augmented trajectories in augmented reality,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2018, pp. 1838–1844

    2018

  12. [20]

    Mimic: In-situ recording and re-use of demonstrations to support robot teleoperation,

    K. Mahadevan, Y . Chen, M. Cakmak, A. Tang, and T. Grossman, “Mimic: In-situ recording and re-use of demonstrations to support robot teleoperation,” in Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology , 2022, pp. 1–13

    2022

  13. [21]

    Scene editing as teleoperation: A case study in 6dof kit assembly,

    Y . Li, S. Agrawal, J.-S. Liu, S. K. Feiner, and S. Song, “Scene editing as teleoperation: A case study in 6dof kit assembly,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022, pp. 4773–4780

    2022

  14. [22]

    Asynchronously assigning, monitoring, and managing assembly goals in virtual reality for high-level robot teleoperation,

    S. Aoyama, J.-S. Liu, P. Wang, S. Jain, X. Wang, J. Xu, S. Song, B. Tversky, and S. Feiner, “Asynchronously assigning, monitoring, and managing assembly goals in virtual reality for high-level robot teleoperation,” in 2024 IEEE Conference Virtual Reality and 3D User Interfaces...

    2024

  15. [23]

    Learning to parse natural language commands to a robot control system,

    C. Matuszek, E. Herbst, L. Zettlemoyer, and D. Fox, “Learning to parse natural language commands to a robot control system,” in Experimen- tal robotics: the 13th international symposium on experimental robotics. Springer, 2013, pp. 403–415

    2013

  16. [24]

    Robots that use language,

    S. Tellex, N. Gopalan, H. Kress-Gazit, and C. Matuszek, “Robots that use language,” Annual Review of Control, Robotics, and Autonomous Systems, vol. 3, no. 1, pp. 25–55, 2020

    2020

  17. [25]

    Inner monologue: Embod- ied reasoning through planning with language models,

    W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y . Chebotaret al., “Inner monologue: Embod- ied reasoning through planning with language models,” in Proceedings of Machine Learning Research , vol. 205, 2023, pp. 1769–1782

    2023

  18. [26]

    Robots that ask for help: Uncer- tainty alignment for large language model planners,

    A. Z. Ren, A. Dixit, A. Bodrova, S. Singh, S. Tu, N. Brown, P. Xu, L. Takayama, F. Xia, J. Varley et al., “Robots that ask for help: Uncer- tainty alignment for large language model planners,” in Proceedings of the 7th Conference on Robot Learning , 2023

    2023

  19. [27]

    Distilling and retrieving generalizable knowledge for robot manipulation via language corrections,

    L. Zha, Y . Cui, L.-H. Lin, M. Kwon, M. G. Arenas, A. Zeng, F. Xia, and D. Sadigh, “Distilling and retrieving generalizable knowledge for robot manipulation via language corrections,” in 2024 IEEE International Conference on Robotics and Automation (ICRA) , 2024

    2024

  20. [28]

    Learning to learn faster from human feedback with language model predictive control,

    J. Liang, F. Xia, W. Yu, A. Zeng, M. G. Arenas, M. Attarian, M. Bauza, M. Bennice, A. Bewley, A. Dostmohamed et al. , “Learning to learn faster from human feedback with language model predictive control,” arXiv preprint arXiv:2402.11450 , 2024

    2024 arXiv

  21. [29]

    Code as policies: Language model programs for embodied control,

    J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng, “Code as policies: Language model programs for embodied control,” in 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 9493–9500

    2023

  22. [30]

    Progprompt: Generating situated robot task plans using large language models,

    I. Singh, V . Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg, “Progprompt: Generating situated robot task plans using large language models,” in 2023 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2023, pp. 11 523–11 530

    2023

  23. [31]

    Ok-robot: What really matters in integrating open-knowledge models for robotics,

    P. Liu, Y . Orru, C. Paxton, N. M. M. Shafiullah, and L. Pinto, “Ok-robot: What really matters in integrating open-knowledge models for robotics,” arXiv preprint arXiv:2401.12202 , 2024

    2024 arXiv

  24. [32]

    Generative expressive robot behaviors using large language models,

    K. Mahadevan, J. Chien, N. Brown, Z. Xu, C. Parada, F. Xia, A. Zeng, L. Takayama, and D. Sadigh, “Generative expressive robot behaviors using large language models,” in Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction , 2024, pp. 482–491

    2024

  25. [33]

    Rt-1: Robotics transformer for real-world control at scale,

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu et al. , “Rt-1: Robotics transformer for real-world control at scale,” in Robotics: Science and Systems , 2023

    2023

  26. [34]

    Octo: An open-source generalist robot policy,

    O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu et al. , “Octo: An open-source generalist robot policy,” in Robotics: Science and Systems , 2024

    2024

  27. [35]

    Open- vla: An open-source vision-language-action model,

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi et al. , “Open- vla: An open-source vision-language-action model,” arXiv preprint arXiv:2406.09246, 2024

    2024 arXiv

  28. [36]

    Code3: A system for end-to-end pro- gramming of mobile manipulator robots for novices and experts,

    J. Huang and M. Cakmak, “Code3: A system for end-to-end pro- gramming of mobile manipulator robots for novices and experts,” in Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction, 2017, pp. 453–462

    2017

  29. [37]

    Design and evaluation of a rapid programming system for service robots,

    J. Huang, T. Lau, and M. Cakmak, “Design and evaluation of a rapid programming system for service robots,” in 2016 11th ACM/IEEE In- ternational Conference on Human-Robot Interaction (HRI) . IEEE, 2016, pp. 295–302

    2016

  30. [38]

    Evaluating coblox: A comparative study of robotics programming environments for adult novices,

    D. Weintrop, A. Afzal, J. Salac, P. Francis, B. Li, D. C. Shepherd, and D. Franklin, “Evaluating coblox: A comparative study of robotics programming environments for adult novices,” in Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems , 2018, pp. 1–12

    2018

  31. [39]

    Roboflow: A flow- based visual programming language for mobile manipulation tasks,

    S. Alexandrova, Z. Tatlock, and M. Cakmak, “Roboflow: A flow- based visual programming language for mobile manipulation tasks,” in 2015 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2015, pp. 5537–5544

    2015

  32. [40]

    Authoring and verifying human-robot interactions,

    D. Porfirio, A. Saupp ´e, A. Albarghouthi, and B. Mutlu, “Authoring and verifying human-robot interactions,” in Proceedings of the 31st annual acm symposium on user interface software and technology , 2018, pp. 75–86

    2018

  33. [41]

    Programar: Augmented reality end-user robot programming,

    B. Ikeda and D. Szafir, “Programar: Augmented reality end-user robot programming,” ACM Transactions on Human-Robot Interaction , vol. 13, no. 1, pp. 1–20, 2024

    2024

  34. [42]

    Aug- mented reality and robotics: A survey and taxonomy for ar-enhanced human-robot interaction and robotic interfaces,

    R. Suzuki, A. Karim, T. Xia, H. Hedayati, and N. Marquardt, “Aug- mented reality and robotics: A survey and taxonomy for ar-enhanced human-robot interaction and robotic interfaces,” in Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems , 2022, pp. 1–33

    2022

  35. [43]

    Projection-based augmented reality interface for robot grasping tasks,

    L. Gong, S. Ong, and A. Nee, “Projection-based augmented reality interface for robot grasping tasks,” in Proceedings of the 2019 4th International Conference on Robotics, Control and Automation , 2019, pp. 100–104

    2019

  36. [44]

    Ghostar: A time-space editor for embodied authoring of human-robot collaborative task with augmented reality,

    Y . Cao, T. Wang, X. Qian, P. S. Rao, M. Wadhawan, K. Huo, and K. Ramani, “Ghostar: A time-space editor for embodied authoring of human-robot collaborative task with augmented reality,” in Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology, ...

    2019

  37. [45]

    Sketch and run: a stroke-based interface for home robots,

    D. Sakamoto, K. Honda, M. Inami, and T. Igarashi, “Sketch and run: a stroke-based interface for home robots,” in Proceedings of the SIGCHI conference on human factors in computing systems , 2009, pp. 197– 200

    2009

  38. [46]

    Sketching robot programs on the fly,

    D. Porfirio, L. Stegner, M. Cakmak, A. Saupp ´e, A. Albarghouthi, and B. Mutlu, “Sketching robot programs on the fly,” in Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction, 2023, pp. 584–593

    2023

  39. [47]

    Situated tangible robot programming,

    Y . S. Sefidgar, P. Agarwal, and M. Cakmak, “Situated tangible robot programming,” in Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction , 2017, pp. 473–482

    2017

  40. [48]

    Pati: a projection-based augmented table- top interface for robot programming,

    Y . Gao and C.-M. Huang, “Pati: a projection-based augmented table- top interface for robot programming,” in Proceedings of the 24th international conference on intelligent user interfaces, 2019, pp. 345– 355

    2019

  41. [49]

    Figaro: A tabletop authoring environment for human-robot interaction,

    D. J. Porfirio, L. Stegner, M. Cakmak, A. Saupp ´e, A. Albarghouthi, and B. Mutlu, “Figaro: A tabletop authoring environment for human-robot interaction,” in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems , 2021, pp. 1–15

    2021

  42. [50]

    Alchemist: Llm-aided end-user development of robot applications,

    U. B. Karli, J.-T. Chen, V . N. Antony, and C.-M. Huang, “Alchemist: Llm-aided end-user development of robot applications,” in Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction, 2024, pp. 361–370

    2024

  43. [51]

    Cocobo: Exploring large language models as the engine for end-user robot programming,

    Y . Ge, Y . Dai, R. Shan, K. Li, Y . Hu, and X. Sun, “Cocobo: Exploring large language models as the engine for end-user robot programming,” in IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), 2024

    2024

  44. [52]

    Robotist: Interactive situated tangible robot programming,

    Y . S. Sefidgar, T. Weng, H. Harvey, S. Elliott, and M. Cakmak, “Robotist: Interactive situated tangible robot programming,” in Proceedings of the 2018 ACM Symposium on Spatial User Interaction , 2018, pp. 141– 149

    2018

  45. [53]

    Vipo: Spatial-visual programming with functions for robot-iot workflows,

    G. Huang, P. S. Rao, M.-H. Wu, X. Qian, S. Y . Nof, K. Ramani, and A. J. Quinn, “Vipo: Spatial-visual programming with functions for robot-iot workflows,” in Proceedings of the 2020 CHI conference on human factors in computing systems , 2020, pp. 1–13

    2020

  46. [54]

    Bodystorming human-robot interactions,

    D. Porfirio, E. Fisher, A. Saupp ´e, A. Albarghouthi, and B. Mutlu, “Bodystorming human-robot interactions,” in Proceedings of the 32nd annual ACM symposium on user Interface software and technology , 2019, pp. 479–491

    2019

  47. [55]

    Robocasa: Large-scale simulation of ev- eryday tasks for generalist robots,

    S. Nasiriany, A. Maddukuri, L. Zhang, A. Parikh, A. Lo, A. Joshi, A. Mandlekar, and Y . Zhu, “Robocasa: Large-scale simulation of ev- eryday tasks for generalist robots,” arXiv preprint arXiv:2406.02523 , 2024

    2024 arXiv

  48. [56]

    Robot see robot do: Imitating articulated object manip- ulation with monocular 4d reconstruction,

    J. Kerr, C. M. Kim, M. Wu, B. Yi, Q. Wang, K. Goldberg, and A. Kanazawa, “Robot see robot do: Imitating articulated object manip- ulation with monocular 4d reconstruction,” in 8th Annual Conference on Robot Learning , 2024

    2024

  49. [57]

    Joint discovery of object states and manipulation actions,

    J.-B. Alayrac, I. Laptev, J. Sivic, and S. Lacoste-Julien, “Joint discovery of object states and manipulation actions,” in Proceedings of the IEEE International Conference on Computer Vision , 2017, pp. 2127–2136

    2017

  50. [58]

    Under- standing 3d object articulation in internet videos,

    S. Qian, L. Jin, C. Rockwell, S. Chen, and D. F. Fouhey, “Under- standing 3d object articulation in internet videos,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 1599–1609

    2022

  51. [59]

    Evaluating real-world robot manipulation policies in simulation,

    X. Li, K. Hsu, J. Gu, K. Pertsch, O. Mees, H. R. Walke, C. Fu, I. Lunawat, I. Sieh, S. Kirmani et al. , “Evaluating real-world robot manipulation policies in simulation,”arXiv preprint arXiv:2405.05941, 2024

    2024 arXiv

  52. [60]

    Instructpix2pix: Learning to follow image editing instructions,

    T. Brooks, A. Holynski, and A. A. Efros, “Instructpix2pix: Learning to follow image editing instructions,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 18 392–18 402

    2023

  53. [61]

    Zero-shot robotic manipulation with pretrained image- editing diffusion models,

    K. Black, M. Nakamoto, P. Atreya, H. Walke, C. Finn, A. Kumar, and S. Levine, “Zero-shot robotic manipulation with pretrained image- editing diffusion models,” arXiv preprint arXiv:2310.10639 , 2023

    2023 arXiv

  54. [62]

    Image inpainting: A review,

    O. Elharrouss, N. Almaadeed, S. Al-Maadeed, and Y . Akbari, “Image inpainting: A review,” Neural Processing Letters , vol. 51, pp. 2007– 2028, 2020

    2007

  55. [63]

    Scaling open-vocabulary object detection,

    M. Minderer, A. Gritsenko, and N. Houlsby, “Scaling open-vocabulary object detection,” Advances in Neural Information Processing Sys- tems, vol. 36, 2024

    2024

  56. [64]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Loet al., “Segment anything,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 4015–4026

    2023

  57. [65]

    Nasa-task load index (nasa-tlx); 20 years later,

    S. G. Hart, “Nasa-task load index (nasa-tlx); 20 years later,” in Proceed- ings of the human factors and ergonomics society annual meeting , vol. 50, no. 9. Sage publications Sage CA: Los Angeles, CA, 2006, pp. 904–908

    2006

  58. [66]

    An empirical evaluation of the system usability scale,

    A. Bangor, P. T. Kortum, and J. T. Miller, “An empirical evaluation of the system usability scale,” Intl. Journal of Human–Computer Interaction, vol. 24, no. 6, pp. 574–594, 2008

    2008

  59. [67]

    Contact- graspnet: Efficient 6-dof grasp generation in cluttered scenes,

    M. Sundermeyer, A. Mousavian, R. Triebel, and D. Fox, “Contact- graspnet: Efficient 6-dof grasp generation in cluttered scenes,” in 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021, pp. 13 438–13 444

    2021

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.