{"id":"a0636918-de6d-4be5-a809-84a18ab0f851","arxiv_id":"2503.15500","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Users who directly manipulated images in the ImageInThat timeline interface generated robot instructions faster and with higher confidence than with a text-based method, in a ten-participant kitchen-task study.","lead":"ImageInThat lets people tell a robot what to do by dragging objects and toggling fixtures in timeline-style pictures of a kitchen. In a ten-person study, users made instructions 64.8% faster and preferred it over typing text steps, suggesting images can reduce ambiguity in robot commanding.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Task-level paired t-test (t(37)) treats 38 observations from 10 participants as independent; the speed claim needs reanalysis with participant/task as random effects.","rationale":"The paper makes a clear comparative claim: image manipulation is faster than text for instructing robots. The strongest evidence for this is the reported time difference. However, the statistical test used to support it is questionable. The reported t(37) implies 38 observations, matching task-level data from 10 participants over 4 tasks. This is pseudo-replication: each participant contributes four correlated observations, and the same task appears across participants. A paired t-test assumes independent pairs; this assumption is violated, so the p-value is likely too small and the confidence intervals too narrow. This is a more fundamental issue than the text-baseline representativeness, because it affects whether the data demonstrate an effect at all, not just how the effect generalizes. The authors do acknowledge the baseline limitation in Section VII, but they do not acknowledge the statistical issue. I would request a reanalysis before relying on the headline claim. The qualitative findings (confidence, preference, SUS) are less affected because SUS was analyzed at participant level (t(9)) and confidence/errors are secondary. If the reanalysis confirms the effect, the paper is a solid conditional accept; if not, the claim should be downgraded. Therefore my concern does not change the reader's CONDITIONAL verdict, but it identifies a different primary weakness.","tokens_in":15925,"tokens_out":5361,"duration_ms":54548,"concrete_test":"Request or extract the raw completion times for each participant and task, then: (1) compute per-participant mean time per condition and run a paired t-test (N=10); (2) fit a linear mixed-effects model with condition as fixed effect and random intercepts for participant and task (or a Wilcoxon signed-rank test on per-participant means as a nonparametric check). Report the estimated condition effect and 95% CI. If the effect remains significant and the relative reduction stays above ~50%, the concern is resolved; if not, the headline speed advantage is not supported by the current statistical evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central speed claim ('Participants were faster... t(37) = -8.96, p <0.001') is computed on task-level data: 10 participants × 4 tasks = 40 task pairs, minus 2 corrupted = 38. A paired t-test on these pairs violates independence because each participant contributes four non-independent observations and each task is reused across participants. The paper says paired t-tests were used 'to account for repeated measures,' but the reported degrees of freedom show the analysis did not account for the nesting. The correct analysis is either a paired t-test on per-participant mean scores (N=10) or a mixed-effects model with random intercepts for participant and task. This is an internal-validity threat to the headline result: if the effect does not survive the appropriate analysis, the claim of faster instruction is not established, regardless of the acknowledged limitations of the text baseline.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ImageInThat, a prototype that lets users instruct a robot by directly manipulating images of the robot's environment in a timeline-style editor, supported by LLM-based captioning, language-to-image editing, and step prediction. The authors report a within-subjects user study (N=10) comparing ImageInThat to a text-based instruction interface across four simulated kitchen tasks, finding a 64.8% reduction in task completion time, higher confidence, higher SUS scores, and no significant overall error difference. The paper also presents a case study translating user-generated image instructions into robot policy code and executing them on a physical robot arm.","tokens_in":16062,"tokens_out":7168,"duration_ms":69338,"significance":"If the empirical findings are valid, ImageInThat offers a compelling alternative paradigm for end-user robot instruction, with concrete grounding in images and potential to reduce ambiguity relative to typed text. The work is significant for HRI and end-user programming: it combines direct manipulation, timeline feedback, and foundation models in a functional prototype, and it includes transparent limitations, bootstrapped confidence intervals, an oracle-based error metric, and released code and supplementary material. However, the central speed claim rests on a task-level paired t-test that violates independence given the nested design, so a reanalysis is necessary before the headline result can be accepted. The comparison baseline is also self-admittedly a worst case for text, so the paradigm-level claim should be scoped accordingly.","major_comments":[{"comment":"The headline speed comparison is reported as t(37) = −8.96, p <0.001 on 38 task-level observations (10 participants × 4 tasks, minus two corrupted). Because each participant contributes up to four observations and each task is reused across participants, the observations are not independent; a paired t-test on task-level data does not account for the repeated-measures structure, contrary to the claim in Section V that paired t-tests were used 'to account for repeated measures.' Please reanalyze the completion-time, error, and confidence results either as paired t-tests on per-participant means (N=10) or with a mixed-effects model including random intercepts for participant and task, and report the resulting effect sizes, confidence intervals, and p-values. If the speed effect does not survive this reanalysis, the abstract's '64.8% less time' claim is not supported by the current analysis.","section":"V"},{"comment":"The authors acknowledge in Section VII that the text condition required step-by-step typed instructions, making it a 'worst-case scenario for the text-based method' while ImageInThat is a 'best-case scenario.' Because the abstract and conclusion state the result as a general comparison ('participants were faster with ImageInThat'), the current study does not establish that image manipulation is faster than natural-language instruction generally. Please either add a more representative language condition (e.g., speech input, or an LLM-based decomposition of high-level goals) or consistently qualify every claim, including the abstract, to refer to the specific text-based interface tested. Without this, the paradigm-level conclusion overreaches the evidence.","section":"VII"}],"minor_comments":[{"comment":"The two sentences reporting extraneous and inefficient steps both give t(37) = 1.35 and report the same means (M = 0.29 vs. M = 0.26), but the condition order is reversed; please correct the typo and define the sign convention for the paired differences so readers can verify the direction.","section":"V"},{"comment":"Section IV describes the number of proposed next-step actions as 'a system parameter' but does not state the value used in the study; please report the default and any sensitivity to it.","section":"IV"},{"comment":"The oracle-based error coding was performed by two experimenters, but no inter-rater reliability measure is reported; please report agreement (e.g., Cohen's kappa) on a subset of tasks.","section":"V"},{"comment":"The manuscript reports several p-values only as inequalities (p < 0.05, p > 0.05) and does not state whether tests are one- or two-tailed; please report exact p-values and test direction.","section":"V"},{"comment":"In Section VI, 'the robot must be able to execute them, This could be accomplished' has a comma splice; also, 'we illustrate a case study' would be more naturally phrased as 'we present a case study.'","section":"VI"},{"comment":"The text callout 'Figure 6 provides a breakdown of participants' completion time' refers only to the middle panel, but the figure contains three panels (errors, time, TLX); please label the panels and reference them specifically in the text.","section":"V"}],"recommendation":"major_revision","confidential_remarks":"The statistical concern is the main obstacle: the task-level paired t-test with t(37) does not respect the nested structure of the data, so the headline speed claim needs a proper reanalysis. Given that the mean difference is large (110.8 vs. 363.9 seconds), the effect may well survive a mixed-model reanalysis, but the paper as written does not provide the necessary evidence. I would ask the authors to supply the reanalyzed values and to make the scope of their claims consistent with the chosen baseline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one. ImageInThat is a real system, not a mockup: users manipulate a 2D image timeline to generate robot instructions, with LLM autocaptioning, language-to-image steps, and goal prediction. The paper ships code and includes a physical robot demonstration translating the images to skill code (7-10/10 on the harder tasks). That alone puts it ahead of many HRI papers.\n\nThe user study is competently designed: within-subjects, counterbalanced, four kitchen tasks, oracle-based error coding, and the qualitative findings (confidence, SUS, preference) are clear. I believe the direction—image manipulation reduces ambiguity and helps users track long-horizon tasks—is plausible and likely real.\n\nBut two things keep me from accepting the speed claim as stated. First, the authors admit in Section VII that the text baseline is the worst case for language: users had to type every low-level step, with no speech and no LLM decomposition. That is a legitimate design choice for a controlled comparison, but it means the 64.8% time reduction is specific to that baseline, not a general property of image manipulation. Second—and this is the bigger problem—the headline t-test is computed on 38 task-level observations from 10 participants. A paired t-test with t(37) treats each task as independent; it ignores the nesting of tasks within participants and the reuse of the same tasks across participants. The paper says 'paired t-test... to account for repeated measures,' but the degrees of freedom show it didn't. The correct analysis is per-participant means or a mixed model with random intercepts for participant and task. The effect is large (110.8 vs 363.9 seconds), so it may survive, but the paper needs to show that.\n\nMinor points: N=10 is small but typical for this kind of study; the image-to-code evaluation is translation-only, not full execution; and the real-robot case study is a single demonstration. None of these are fatal.\n\nWho should read it: people working on end-user robot programming, direct manipulation, and human-robot instruction interfaces. The system concept is worth engaging with even if the speed claim needs reanalysis.\n\nRecommendation: send it to peer review. The engineering is solid, the limitation discussion is honest, and the statistical issue is fixable in revision. I'd want to see the mixed-model reanalysis and the raw data before accepting the abstract's speed claim.","headline":"A well-built system and an honest write-up, but the headline speed claim rests on a self-admittedly unfair text baseline and a t-test that ignores repeated measures.","tokens_in":16603,"tokens_out":2515,"would_cite":true,"duration_ms":25148,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that directly dragging objects in a camera image lets users create robot instructions 64.8 percent faster than typing step-by-step text, with higher confidence and no more errors.","keywords":["direct manipulation","robot instruction","end-user robot programming","image-based interfaces","timeline interface","human-robot interaction","kitchen manipulation tasks","language-based instruction"],"falsifier":"Run the same four kitchen tasks with a speech-based or LLM-decomposed text condition; if completion times become statistically indistinguishable from ImageInThat, the central faster-than-text claim no longer generalizes to natural language instruction.","tokens_in":15728,"feed_emoji":"🖼️","tokens_out":4398,"duration_ms":40095,"temperature":0.7,"pith_summary":"ImageInThat proposes that people can instruct robots more effectively by directly dragging and clicking objects in a camera image of the robot's environment than by typing step-by-step text commands. In a within-subjects study with ten participants performing four kitchen manipulation tasks, the image-based interface took 110.8 seconds on average versus 363.9 seconds for the text-based method, a 64.8 percent reduction. Participants also reported higher confidence that the robot understood their instructions, rated usability higher, reported lower workload, and preferred the image method. The paper argues that images are intrinsically grounded in the robot's environment, so direct manipulation removes the ambiguity of referring to specific objects and placements in language. It also demonstrates that the resulting image instructions can be translated into robot policy code and executed on a physical arm.","feed_headline":"Drag images, don't type: robot instructions 64.8% faster","feed_subtitle":"A timeline interface where users move objects in photos beat typed step lists in speed, confidence, and usability.","key_machinery":"The carrying mechanism is a timeline-style editor over a segmented visual representation of the robot's environment. Each object is a draggable mask overlaid on background images that encode fixture states (e.g., cabinet open or closed); every user manipulation appends a thumbnail step to the timeline, and the system highlights what changed between consecutive steps to keep the state evolution legible. On top of this, an LLM generates captions for each manipulation, accepts text edits that modify the image, and proposes goal locations and plausible next steps. The same representation is what makes the paradigm work as an instruction format: the image itself is the concrete, grounded specification, and the supporting models translate it into captions or policy code for execution.","core_discovery":"The paper's central claim is that direct manipulation of images is a viable and preferable paradigm for conveying step-by-step instructions to a robot. ImageInThat instantiates this by letting users select objects and fixtures in a photographic view and drag them to new locations or toggle their states, with each change automatically inserted as a step in a visual timeline. The system couples these manipulations with automatic captions, language-based image editing, and goal prediction, so the user always sees the environment state that the robot would achieve. The user study found the image condition was 64.8 percent faster than the text condition, with significantly higher confidence, usability, and preference, and no overall error increase; one notable difference was that the text condition produced significantly more missing steps. The paper frames image-based instructions as a middle ground between immediate language commands and traditional end-user programming, supporting both near-real-time commanding and longer-horizon task specification.","pith_inferences":["The reported speed advantage is measured against typed text only; the paper itself grants that speech-based or LLM-decomposed natural language could be a stronger baseline, so the headline 'images beat language' should be read as 'images beat typed low-level steps' until such baselines are tested.","Image instructions may serve as a shared grounding format across robot policies: goal-image-conditioned policies and vision-language-action models could consume user-edited images directly, bypassing error-prone language grounding.","The interaction trace (which objects moved, when, and to where) is a natural dataset for learning user preferences, such as always placing heavy dishes on lower shelves, something the paper mentions only as future work.","A direct comparison to speech or to demonstration-based programming would clarify whether the benefit comes from the image modality itself or from the absence of typing."],"forward_implications":["People can specify a multi-step kitchen task in about a third of the time it takes to type the equivalent step list, with no loss of correctness.","Users trust image-based instructions more: they rated confidence and usability higher and reported lower workload than with text.","The timeline representation supports the four components of robot instruction: capturing intent, translating it, presenting it for confirmation, and executing it.","Image instructions produced with the interface can be turned into executable policy code for a physical robot arm, at least for tasks involving clear object identities.","Because missing steps were significantly rarer with images, the format may particularly help in longer-horizon tasks where users must keep track of object states."],"supporting_citations":[{"why":"Supplies the direct manipulation concept that the image-instruction paradigm is built on.","marker":"[4]"},{"why":"Identifies the difficulty of capturing user intent in end-user robot programming that ImageInThat targets.","marker":"[3]"},{"why":"Provides the code-as-policies approach used to translate image instructions into executable robot code.","marker":"[29]"},{"why":"Represents natural-language robot commanding that the work positions against.","marker":"[1]"},{"why":"Shows language-conditioned policies that typically take single instructions, motivating the single-object step design.","marker":"[35]"},{"why":"Provides the Robocasa simulated kitchen environments used to generate study images.","marker":"[55]"},{"why":"Open-vocabulary detector used to find objects and fixtures in the environment.","marker":"[63]"},{"why":"Segment Anything produces the object masks that make direct manipulation possible.","marker":"[64]"},{"why":"Contact-GraspNet generates candidate grasps in the physical robot execution case study.","marker":"[67]"}],"fun_headline_variants":["Drag photos to program robots: 64.8% faster than typing","Image dragging beats text for robot commands, user study finds","Manipulate images, not words: faster robot instructions","Teach robots by dragging images: interface wins in speed","Robot instructions via image drag: users faster, prefer it"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison treats typed, step-by-step text as the stand-in for natural-language robot instruction; if a more natural language interface (speech or LLM decomposition) is used, the speed advantage may shrink or vanish.","fun_headline_variants_meta":{"raw":{"variants":["Drag photos to program robots: 64.8% faster than typing","Image dragging beats text for robot commands, user study finds","Manipulate images, not words: faster robot instructions","Teach robots by dragging images: interface wins in speed","Robot instructions via image drag: users faster, prefer it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000627,"raw_usage":{"total_tokens":2880,"prompt_tokens":905,"completion_tokens":1975,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":1892}},"tokens_in":521,"tokens_out":1975,"duration_ms":12652,"temperature":1.0,"reasoning_tokens":1892,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:48:25.030194+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same four kitchen tasks with a speech-based or LLM-decomposed text condition; if completion times become statistically indistinguishable from ImageInThat, the central faster-than-text claim no longer generalizes to natural language instruction.","supporting_citations":[{"cited_title":"Direct manipulation: A step beyond programming languages,","cited_arxiv_id":null,"evidence_quote":"Supplies the direct manipulation concept that the image-instruction paradigm is built on."},{"cited_title":"A survey on end-user robot programming,","cited_arxiv_id":null,"evidence_quote":"Identifies the difficulty of capturing user intent in end-user robot programming that ImageInThat targets."},{"cited_title":"Scaling open-vocabulary object detection,","cited_arxiv_id":null,"evidence_quote":"Open-vocabulary detector used to find objects and fixtures in the environment."},{"cited_title":"Contact- graspnet: Efficient 6-dof grasp generation in cluttered scenes,","cited_arxiv_id":null,"evidence_quote":"Contact-GraspNet generates candidate grasps in the physical robot execution case study."}],"review_version":1}