ReStory: VLM-augmentation of Social Human-Robot Interaction Datasets

Fanjun Bu; Wendy Ju

REVIEW 4 major objections 5 minor 24 references

ReStory: VLM-augmentation of Social Human-Robot Interaction Datasets

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ReStory, a vision-language pipeline, can reorder a second person's actions into the sequence of a reference interaction storyboard, producing new human-interpretable HRI scenarios.

desk verdict A genuinely novel pipeline for storyboard-based HRI data augmentation, but the validation cannot yet carry the claim that the synthesized storyboards capture the target interaction pattern. read the letter →

arxiv 2412.20826 v1 pith:VTWZ6NEG submitted 2024-12-30 cs.RO cs.HC

Fanjun Bu , Wendy Ju This is my paper · ORCID

classification cs.ROcs.HC

keywords Synthetichuman-robotinteractionDataaugmentationVision-languagemodelsStoryboardingEthnomethodologySentence-BERTFrameretrieval

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Human-robot interaction researchers lack internet-scale datasets, so they cannot easily train data-hungry models. ReStory proposes to augment existing in-the-wild HRI footage by re-splicing one participant's actions into the storyboard of another interaction. The paper claims that, given an ethnographer-crafted storyboard and a second video, a vision-language model can caption both, and a semantic similarity metric can find the video frames that best match each storyboard keyframe, yielding a new storyboard of a different person performing the same interaction pattern. A validation with seven annotators suggests the synthesized storyboards convey the defining semantic signatures of the base patterns, such as avoidance in the driveby scenario. The approach still requires human supervision, but it offers a new way to stretch scarce interaction data.

What carries the argument

The central machinery is the semantic frame-retrieval loop: a two-prompt VLM captioning scheme (posture first, then interaction context conditioned on posture and previous context) combined with SBERT embedding and a weighted cosine-similarity score (context weight 0.8, pose weight 0.2) that selects the best-matching frame from a source video for each storyboard keyframe. This loop aligns the source person's atomic actions with the reference sequence's semantic content, ignoring pixel-level differences such as left- vs right-handed waving, so that the new storyboard preserves the high-level interaction pattern while introducing fresh visual variation.

What would settle it

Use ReStory with a source video in which the robot approaches the person from a different side or at a different speed than in the reference storyboard, then ask annotators to name the interaction pattern; if the generated storyboard no longer conveys the reference pattern, the key assumption that robot behavior is constant is violated.

Watch

Extended reading notes

Core claim

ReStory rests on the idea that an interaction can be compressed into a storyboard: a sequence of keyframes whose captions describe the postures and contextual actions of the people involved. The method takes an existing EMCA storyboard as a reference and a different interaction video as a source. For each storyboard frame and for two-frames-per-second samples of the source video, ReStory prompts a vision-language model to produce a posture caption (P1) and, conditioned on that posture and the previous frame's context, an interaction-context caption (P2). It then computes the weighted cosine similarity between SBERT embeddings of the captions, with context weighted more heavily than posture (α = 0.2), and selects for each storyboard frame the source frame with the highest similarity. The selected frames are stitched, with the robot's ego-motion between consecutive frames given as text, to form a new storyboard in which the source person appears to perform the reference interaction pattern. The paper's claim is that this produces human-interpretable scenarios that preserve the semantic-level signatures of the base sequences, as suggested by the narration study where annotators recognized the defining features of the driveby, offer-and-release, and ask-and-receive patterns at rates comparable to the original storyboards.

Load-bearing premise

ReStory assumes that the robot's motion in the generated storyboard matches the robot's motion in the reference storyboard; if the robot approaches or moves differently, the spliced human frames no longer tell the same interaction story.

Editorial extensions

If this is right

Existing HRI datasets can be multiplied by combining each interaction pattern with different participants' footage, generating new storyboard episodes without additional real-world collection.
The prompts and similarity metric are user-defined, so the pipeline can be retargeted to other robot form factors or interaction contexts by changing what the VLM is asked to describe and what similarity weights are used.
The synthesized storyboards can support design ideation, letting interaction designers explore how a different user might experience the same robot behavior.
Because frames are drawn from real footage, personal idiosyncrasies (gesture style, gaze, posture) are preserved even as the action sequence is rearranged, yielding a form of data augmentation that respects individual variation.
Human verification remains necessary: the paper reports that causality violations (two frames swapped) and VLM hallucinations can occur, so the output storyboards are semi-novel rather than fully automatic.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural next step is to apply the same retrieval idea to the robot side, swapping robot behaviors while keeping human footage fixed; this would require solving the paper's weakest assumption about robot behavior being constant and could enable two-party storyboard synthesis.
The effect of the distance confound could be quantified by adding depth as an input feature and measuring whether annotator agreement on the focal action improves; this is a direct testable extension suggested by the paper's own limitation discussion.
ReStory could serve as a data-generation module for training perception models: caption-aligned frame pairs from the same interaction pattern provide weak supervision for pose or action recognition, bypassing the need for manual annotation of new footage.
Because the similarity metric is a weighted combination, the relative importance of posture and context could be tuned per task; for example, interaction phases that hinge on handover might require higher context weight, whereas posture-driven signals like avoidance might need higher pose weight.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The thing to know about this paper is that the core idea is real and worth a look: use a VLM to caption frames, then use SBERT similarity to retrieve frames from a different video that match a reference EMCA storyboard, and reassemble them into a new storyboard. I have not seen that exact combination before, and for a research community starved of interaction data, it is a sensible angle. The paper is also honest about being a proof-of-concept and openly lists its limitations: distance perception, causality violations, VLM hallucination, and the need for human supervision. That transparency earns credit. What the paper does well: the pipeline is described clearly enough to reproduce in broad strokes, the choice of prompts is motivated, and the framing around EMCA storyboards as templates is a genuine contribution. The authors are not over-claiming in the abstract; they say 'has the potential' and 'human-interpretable,' which is appropriate. The soft spots are mostly in the evaluation. There is no control condition, so the fact that annotators could narrate the synthesized storyboards does not tell you ReStory's similarity-based retrieval is doing the work; random frame selection might do just as well. The annotators are seven people from the same research group with dataset access, which biases them toward recognizing the patterns. No code, data, or exact prompts are released, so the results are not independently checkable. The hand-set alpha (0.2) and frame sampling rate have no sensitivity analysis. One storyboard needed manual frame reordering, which the paper reports honestly but which weakens the 'automatic synthesis' story. The stress-test note about robot behavior is the most serious issue, and it holds up. Section 3.2 explicitly assumes the robot's behavior in the new storyboard matches the reference. In the demo, the robot is tele-operated, and the source videos are chosen because they show different interaction patterns. So the robot's approach, stop, and pass trajectories almost certainly differ. The evaluation provides ego-motion text between frames, but it never says whether that text comes from the reference storyboard or the source video. Either way, the synthesized storyboard may be mixing human frames from one interaction with robot motion from another, and the results cannot be attributed to the image content alone. The paper acknowledges the assumption but never tests it, which is a load-bearing gap. Bottom line: this is a promising proof-of-concept for HRI researchers and interaction designers who work with EMCA-style analysis. It deserves a serious referee, but the validation needs a proper baseline, outside annotators, and an explicit test of the robot-motion assumption before the central claim is supported. I would not cite it yet, but I would read a revised version.

Referee Report

4 major / 5 minor

Summary. ReStory proposes a VLM-driven pipeline for augmenting scarce human-robot interaction (HRI) datasets. Given an ethnographer-crafted reference storyboard and another interaction video, the method captions each reference keyframe and each sampled video frame with two prompts (body posture P1 and interaction context P2), computes weighted SBERT semantic similarities between captions, and retrieves for each reference keyframe the video frame with the highest similarity. The resulting storyboards preserve the high-level action sequence of the reference while introducing a different person's visual appearance and idiosyncratic movements. The authors validate the approach by asking seven in-group researchers to narrate three original storyboards and six synthesized storyboards built from cross-pairings of the three interaction patterns (driveby, offer-and-release, ask-and-receive) from Brown et al.'s trash-barrel robot dataset. They report that annotators frequently, but not always, described the defining signatures of the intended patterns, and they conclude that the preliminary validation demonstrates ReStory's ability to generate storyboards that capture the semantic-level signatures of the base sequences. The paper is explicitly positioned as a proof-of-concept requiring human supervision.

Significance. If the central claim were properly established, ReStory would offer a useful, low-cost way to repurpose existing in-the-wild HRI video into new storyboard-style training or design material, and it would connect EMCA storyboard practices with modern VLM capabilities. The paper's strengths are its clear and modular pipeline description, its transparent enumeration of assumptions (especially the robot-behavior assumption in §3.2), and its candid discussion of limitations in §5.2, including distance perception, causality ordering, and VLM hallucination. However, the reported validation is too weak to support the conclusion as stated: it lacks a control condition, uses a small convenience sample without inter-annotator agreement metrics, and does not verify a load-bearing assumption about robot behavior. The contribution at this stage is therefore a promising proof-of-concept with an unproven efficacy claim, rather than a validated data-augmentation method.

major comments (4)

[§4.1–§4.2] The validation study lacks any control condition. Annotators were shown only storyboards produced by the full ReStory pipeline, so the observed interpretability cannot be attributed to the similarity-based retrieval. A natural control would be randomly sampled frames from the same input videos, or a similarity metric based on random captions, matched for frame count and ordering. Without such a baseline, the counts reported in §4.2 are consistent with the trivial explanation that any frames drawn from a coherent interaction video—regardless of their semantic alignment with the reference storyboard—can be narrated by a motivated annotator.
[§3.2 and §4.1] The paper explicitly states in §3.2 the implicit assumption that the robot's behavior in the new storyboard must match that in the old storyboard, but it never verifies this assumption. The source videos are chosen because they depict different interaction patterns than the reference (e.g., driveby synthesized from offer-and-release footage), and the robot is tele-operated; nothing in the pipeline checks whether the robot's approach, stopping, and passing motion in the source footage matches the reference pattern. The evaluation in §4.1 supplements the storyboards with ego-motion text between consecutive frames, but the paper does not state where this text comes from—the reference storyboard or the source video. If it comes from the reference, the storyboards combine human frames from one interaction with robot motion from another; if it comes from the source, the robot behavior is not the reference behavior and the defining signature of the target interaction may be lost. Either way, the evaluation does not address this load-bearing assumption.
[§4.1–§4.2] The evaluation is circular and under-powered. The prompts P1 and P2 are designed to capture exactly the kind of posture and interaction-context information that defines the three interaction categories, and the retrieval objective is to match the reference storyboard's captions; thus, the synthesized storyboards are constructed to align semantically with the reference at the caption level. Asking annotators to narrate the same three categories then partly measures the consistency of the caption-recaption loop, not the validity of the synthesized interaction. Additionally, seven annotators from the same research group, with no reported inter-annotator agreement measure and no statistical comparison against chance, cannot support the quantitative conclusion in §5.1. The reported counts (e.g., 'six out of seven' for original patterns, 'seven out of fourteen' for synthesized driveby) need a defined success threshold, confidence intervals, or a chance-level baseline to be interpretable.
[§4.2 and §5.1] The results reported in §4.2 do not support the claim in §5.1 that the preliminary validation 'demonstrates that ReStory is able to generate new storyboards that capture the semantic-level signatures of these sequences.' For the synthesized driveby pattern, only 7 of 14 narrations captured the user's avoidance behavior, and for the synthesized ask-and-receive pattern, only 6 of 14 captured robot initiation—close to what would be expected by chance if annotators narrated arbitrary interactions in this setting. The aggregated counts also pool two synthesized storyboards per pattern, obscuring per-storyboard differences. The paper should report results per storyboard, define a target level of agreement before the study, and explicitly compare synthesized against original storyboards on the same metric before drawing the conclusion in §5.1.

minor comments (5)

[§4.1] The annotation task is underspecified: 'narrate frame by frame what happened' and 'report where the storyboard breaks down' are two different tasks, and it is unclear whether annotators were instructed to classify the interaction pattern or simply describe events. The protocol should be described in enough detail to be replicable.
[§4.1] The paper states that annotators were seven researchers within the research group and had access to the dataset but were 'not familiar with the footage.' Please specify whether they were blind to the study hypothesis and whether they had seen the original Brown et al. paper or its storyboards, since this could bias narration toward the expected patterns.
[§3.1 and §5.2] The YOLO cropping of the user from input frames is described in §5.2 as part of the pipeline, but the main pipeline description in §3.1 and Figure 1 do not mention this preprocessing step. Clarify whether cropping was applied in the validation study and how it affects the pose and context captions.
[§4.1] The source of the ego-motion text is not specified—whether it came from robot telemetry, VLM inference, or manual annotation by the authors. This should be stated explicitly because it directly affects the interpretation of the storyboards in the evaluation.
[§3.1] The weighting parameter alpha is fixed at 0.2 without any sensitivity analysis. At minimum, the authors should note that the reported results depend on this choice and that a small sensitivity study would strengthen the claim that the method is robust to the similarity weighting.

Circularity Check

1 steps flagged · score 4.0 of 10

Retrieval objective and semantic-signature evaluation are aligned by construction; human-interpretability test provides partial independent support.

self definitional [Section 3 (Introducing ReStory) and Section 5.1 (Assumptions)]
"For every frame in the given storyboard, we find the image in the footage with the most similar caption to construct a new storyboard. ... The preliminary system validation study demonstrates that ReStory is able to generate new storyboards that capture the semantic-level signatures of these sequences."

The synthesized storyboard is defined as the input frames whose VLM captions are most similar to the reference storyboard captions under the weighted SBERT metric. Therefore, the property 'captures the semantic-level signatures' is encoded in the retrieval objective itself: a frame is selected precisely because its caption matches the reference caption describing the signature. The validation then asks annotators to narrate those same signatures from the spliced frames, so a positive finding partly reflects the construction rule rather than an independent discovery that the interaction pattern generalizes. The claim is not fully circular because human interpretability of the spliced sequence is not guaranteed by caption similarity and was tested separately.

full rationale

ReStory is an honest proof-of-concept: it retrieves, rather than invents, frames and explicitly states its assumptions and limitations. The main circularity concern is that the evaluation target (semantic-level signatures) is the same caption-similarity objective used to build the storyboards; this is a partial self-confirmation. However, the central claim of human-interpretable storyboards does have independent content: naive caption matching could produce incoherent sequences, and the annotator narrations (including emergent 'asked by the robot' readings not present in the original storyboard) provide non-tautological evidence. The self-citations to Brown et al. are data/provenance choices rather than load-bearing uniqueness claims; Brown et al. is a peer-reviewed empirical source and ReStory's pipeline could be applied to other storyboards. The unverified assumption in Section 3.2 that the robot's tele-operated behavior in the source footage matches the reference storyboard is a validity threat to the synthesized scenarios, but it is a stated assumption, not a circular derivation. No fitted parameter is relabeled as a prediction, and no ansatz is smuggled in by citation. Score 4 reflects the partial construction-evaluation alignment while acknowledging the independent human-interpretability assessment.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

ReStory rests on domain assumptions about the transferability of interaction sequences and robot-behavior invariance (Section 3.2), plus hand-set parameters (alpha, sampling rate) that are not sensitivity-analyzed. It introduces no new physical or conceptual entities beyond the pipeline itself.

free parameters (3)

Alpha (pose-context weight) = 0.2
In Section 3.1, the final similarity is defined as alpha*Similarity_p + (1-alpha)*Similarity_c, with alpha set to 0.2 by hand because context was deemed more important than pose. No sensitivity analysis is given, and this weight affects which frames are selected for every synthesized storyboard.
Video frame sampling rate = 2 fps
Section 3 states the input video is sampled at two frames per second 'for practical considerations' and notes the rate is subject to change. This defines the candidate frame pool for retrieval and thus influences which frames can appear in the new storyboard.
Prompt templates P1/P2 and storyboard keyframe selection = hand-designed
The captioning prompts are tailored to the trash-transaction context and the reference storyboards come from prior work by the same group. These choices define the semantic space in which similarity is computed and are not varied or released.

assumptions (4)

domain assumption The high-level action sequence of an ethnographer-crafted storyboard is valid and transferable to another person.
Section 3.2 states 'the high-level action sequence is valid since an actual human has performed the same actions in the same order.' This is load-bearing because ReStory transfers the sequence to a different participant.
domain assumption In a given interaction context, human behaviors are constrained to a shared, observable semantic action space.
Section 5.1 states 'people's behaviors are largely constrained in an observable action space for a given interaction context.' This justifies captioning and retrieval across different individuals.
domain assumption The robot's behavior in the synthesized storyboard matches the robot's behavior in the base storyboard.
Section 3.2 states 'we must assume that in the new storyboard, the robot approaches the lady the same way as it approaches the man.' For tele-operated robots this must be separately ensured, and it is a structural premise of the method.
domain assumption VLM captions and SBERT cosine similarity are reliable proxies for semantic similarity of interaction frames.
The entire retrieval depends on captions and embeddings; the authors acknowledge VLM hallucinations as a limitation (Section 5.2) and do not validate caption accuracy or embedding quality against human judgments.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ReStory: VLM-augmentation of Social Human-Robot Interaction Datasets." pith.science (2026). https://pith.science/paper/VTWZ6NEG

@misc{pith2026241220826,
  author       = {Pith},
  title        = {Pith review of: ReStory: VLM-augmentation of Social Human-Robot Interaction Datasets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VTWZ6NEG}},
  note         = {Machine review of arXiv:2412.20826}
}

read the original abstract

Internet-scaled datasets are a luxury for human-robot interaction (HRI) researchers, as collecting natural interaction data in the wild is time-consuming and logistically challenging. The problem is exacerbated by robots' different form factors and interaction modalities. Inspired by recent work on ethnomethodological and conversation analysis (EMCA) in the domain of HRI, we propose ReStory, a method that has the potential to augment existing in-the-wild human-robot interaction datasets leveraging Vision Language Models. While still requiring human supervision, ReStory is capable of synthesizing human-interpretable interaction scenarios in the form of storyboards. We hope our proposed approach provides HRI researchers and interaction designers with a new angle to utilizing their valuable and scarce data.

Figures

Figures reproduced from arXiv: 2412.20826 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 16 canonical work pages

[1]

In: International Conference on Machine Learning

Aher, G.V., Arriaga, R.I., Kalai, A.T.: Using large language models to simu- late multiple humans and replicate human subject studies. In: International Conference on Machine Learning. pp. 337–371. PMLR (2023)

work page 2023
[2]

In: Proceedings of the 23rd ACM International Conference on Intelligent Virtual Agents

Antunes, A., Campos, J., Guimarães, M., Dias, J., Santos, P.A.: Prompting for socially intelligent agents with chatgpt. In: Proceedings of the 23rd ACM International Conference on Intelligent Virtual Agents. pp. 1–9 (2023)

work page 2023
[3]

In: Proceedings of the CHI Conference on Human Factors in Computing Systems

Brown, B., Bu, F., Mandel, I., Ju, W.: Trash in motion: Emergent inter- actions with a robotic trashcan. In: Proceedings of the CHI Conference on Human Factors in Computing Systems. CHI ’24, Association for Comput- ing Machinery, New York, NY, USA (2024),https://doi.org/10.1145/ 3613904.3642610

arXiv 2024
[4]

In: Companion of the 2023 ACM/IEEE International Conference on Human- Robot Interaction

Bu, F., Mandel, I., Lee, W.Y., Ju, W.: Trash barrel robots in the city. In: Companion of the 2023 ACM/IEEE International Conference on Human- Robot Interaction. p. 875–877. HRI ’23, Association for Computing Machin- ery, New York, NY, USA (2023). https://doi.org/10.1145/3568294. 3580206, https://doi.org/10.1145/3568294.3580206

work page arXiv 2023
[5]

In: Proceedings of the CHI Conference on Human Factors in Computing Systems

Duan, P., Warner, J., Li, Y., Hartmann, B.: Generating automatic feedback on ui mockups with large language models. In: Proceedings of the CHI Conference on Human Factors in Computing Systems. pp. 1–20 (2024)

work page 2024
[6]

Goffman, E., et al.: The presentation of self in everyday life. 1959. Garden City, NY259 (2002)

work page 2002
[7]

Human-robot interaction: Evaluation methods and their standardization pp

Jacobs, A., Elprama, S.A., Jewell, C.I.: Evaluating human-robot interaction with ethnography. Human-robot interaction: Evaluation methods and their standardization pp. 269–286 (2020)

work page 2020
[8]

In: Proceedings of the 2008 ACM conference on Computer supported coop- erative work

Kuzuoka, H., Pitsch, K., Suzuki, Y., Kawaguchi, I., Yamazaki, K., Ya- mazaki, A., Kuno, Y., Luff, P., Heath, C.: Effect of restarts and pauses on achieving a state of mutual orientation between a human and a robot. In: Proceedings of the 2008 ACM conference on Computer supported coop- erative work. pp. 201–204 (2008)

work page 2008

Show all 24 references

[9]

In: 2018 IEEE/RSJ International Conference on In- telligent Robots and Systems (IROS)

Lakomkin, E., Zamani, M.A., Weber, C., Magg, S., Wermter, S.: On the robustness of speech emotion recognition for human-robot interaction with deep neural networks. In: 2018 IEEE/RSJ International Conference on In- telligent Robots and Systems (IROS). pp. 854–860. IEEE (2018)

2018
[10]

arXiv preprint arXiv:2405.02957 (2024)

Li, J., Wang, S., Zhang, M., Li, W., Lai, Y., Kang, X., Ma, W., Liu, Y.: Agent hospital: A simulacrum of hospital with evolvable medical agents. arXiv preprint arXiv:2405.02957 (2024)

2024 arXiv
[11]

arXiv preprint arXiv:2402.17753 (2024)

Maharana, A., Lee, D.H., Tulyakov, S., Bansal, M., Barbieri, F., Fang, Y.: Evaluating very long-term conversational memory of llm agents. arXiv preprint arXiv:2402.17753 (2024)

2024 arXiv
[12]

In: Conference on robot learning

Murray, M., Walker, N., Nanavati, A., Alves-Oliveira, P., Filippov, N., Sauppe, A., Mutlu, B., Cakmak, M.: Learning backchanneling behaviors ReStory: VLM-augmentation of Social Human-Robot Interaction Datasets 11 for a social robot via data augmentation from human-human conver...

2022
[13]

In: Proceed- ings of the 36th Annual ACM Symposium on User Interface Software and Technology

Park, J.S., O’Brien, J., Cai, C.J., Morris, M.R., Liang, P., Bernstein, M.S.: Generative agents: Interactive simulacra of human behavior. In: Proceed- ings of the 36th Annual ACM Symposium on User Interface Software and Technology. pp. 1–22 (2023)

2023
[14]

In: Proceedings of the 35th Annual ACM Symposium on User Inter- face Software and Technology

Park, J.S., Popowski, L., Cai, C., Morris, M.R., Liang, P., Bernstein, M.S.: Social simulacra: Creating populated prototypes for social computing sys- tems. In: Proceedings of the 35th Annual ACM Symposium on User Inter- face Software and Technology. pp. 1–18 (2022)

2022
[15]

ACM Transactions on Computer-Human Interaction 30(4), 1–42 (Aug 2023)

Pelikan, H., Hofstetter, E.: Managing Delays in Human-Robot Interac- tion. ACM Transactions on Computer-Human Interaction 30(4), 1–42 (Aug 2023). https://doi.org/10.1145/3569890, https://dl.acm.org/ doi/10.1145/3569890

2023 doi
[16]

In: Proceedings of the 2024 ACM/IEEE Inter- national Conference on Human-Robot Interaction

Pelikan, H.R., Reeves, S., Cantarutti, M.N.: Encountering autonomous robots on public streets. In: Proceedings of the 2024 ACM/IEEE Inter- national Conference on Human-Robot Interaction. pp. 561–571 (2024)

2024
[17]

the first five seconds

Pitsch, K., Kuzuoka, H., Suzuki, Y., Sussenbach, L., Luff, P., Heath, C.: “the first five seconds”: Contingent stepwise entry into an interaction as a means to secure sustained engagement in hri. In: RO-MAN 2009-The 18th IEEE International Symposium on Robot and Human Interact...

2009
[18]

In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 779–788 (2016). https:// doi.org/10.1109/CVPR.2016.91

2016 doi
[19]

arXiv preprint arXiv:1908.10084 (2019)

Reimers, N.: Sentence-bert: Sentence embeddings using siamese bert- networks. arXiv preprint arXiv:1908.10084 (2019)

2019 arXiv
[20]

Journal of big data6(1), 1–48 (2019)

Shorten, C., Khoshgoftaar, T.M.: A survey on image data augmentation for deep learning. Journal of big data6(1), 1–48 (2019)

2019
[21]

Na- ture 633(8031), 779–788 (2024)

Slade, P., Atkeson, C., Donelan, J.M., Houdijk, H., Ingraham, K.A., Kim, M., Kong, K., Poggensee, K.L., Riener, R., Steinert, M., Zhang, J., Collins, S.H.: On human-in-the-loop optimization of human-robot interaction. Na- ture 633(8031), 779–788 (2024)

2024
[22]

In: Proceedings of the 6th conference on Designing Interactive systems

Truong, K.N., Hayes, G.R., Abowd, G.D.: Storyboarding: an empirical de- termination of best practices and effective guidelines. In: Proceedings of the 6th conference on Designing Interactive systems. pp. 12–21 (2006)

2006
[23]

AI & SOCIETY (Jan 2023)

Tuncer, S., Licoppe, C., Luff, P., Heath, C.: Recipient design in hu- man–robot interaction: the emergent assessment of a robot’s com- petence. AI & SOCIETY (Jan 2023). https://doi.org/10.1007/ s00146-022-01608-7, https://doi.org/10.1007/s00146-022-01608-7

2023 doi
[24]

In: Proceedings of the SIGCHI conference on human factors in computing systems

Yamazaki, K., Yamazaki, A., Okada, M., Kuno, Y., Kobayashi, Y., Hoshi, Y., Pitsch, K., Luff, P., Vom Lehn, D., Heath, C.: Revealing gauguin: engag- ing visitors in robot guide’s explanation in an art museum. In: Proceedings of the SIGCHI conference on human factors in computin...

2009

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

In: International Conference on Machine Learning

Aher, G.V., Arriaga, R.I., Kalai, A.T.: Using large language models to simu- late multiple humans and replicate human subject studies. In: International Conference on Machine Learning. pp. 337–371. PMLR (2023)

work page 2023

[2] [2]

In: Proceedings of the 23rd ACM International Conference on Intelligent Virtual Agents

Antunes, A., Campos, J., Guimarães, M., Dias, J., Santos, P.A.: Prompting for socially intelligent agents with chatgpt. In: Proceedings of the 23rd ACM International Conference on Intelligent Virtual Agents. pp. 1–9 (2023)

work page 2023

[3] [3]

In: Proceedings of the CHI Conference on Human Factors in Computing Systems

Brown, B., Bu, F., Mandel, I., Ju, W.: Trash in motion: Emergent inter- actions with a robotic trashcan. In: Proceedings of the CHI Conference on Human Factors in Computing Systems. CHI ’24, Association for Comput- ing Machinery, New York, NY, USA (2024),https://doi.org/10.1145/ 3613904.3642610

arXiv 2024

[4] [4]

In: Companion of the 2023 ACM/IEEE International Conference on Human- Robot Interaction

Bu, F., Mandel, I., Lee, W.Y., Ju, W.: Trash barrel robots in the city. In: Companion of the 2023 ACM/IEEE International Conference on Human- Robot Interaction. p. 875–877. HRI ’23, Association for Computing Machin- ery, New York, NY, USA (2023). https://doi.org/10.1145/3568294. 3580206, https://doi.org/10.1145/3568294.3580206

work page arXiv 2023

[5] [5]

In: Proceedings of the CHI Conference on Human Factors in Computing Systems

Duan, P., Warner, J., Li, Y., Hartmann, B.: Generating automatic feedback on ui mockups with large language models. In: Proceedings of the CHI Conference on Human Factors in Computing Systems. pp. 1–20 (2024)

work page 2024

[6] [6]

Goffman, E., et al.: The presentation of self in everyday life. 1959. Garden City, NY259 (2002)

work page 2002

[7] [7]

Human-robot interaction: Evaluation methods and their standardization pp

Jacobs, A., Elprama, S.A., Jewell, C.I.: Evaluating human-robot interaction with ethnography. Human-robot interaction: Evaluation methods and their standardization pp. 269–286 (2020)

work page 2020

[8] [8]

In: Proceedings of the 2008 ACM conference on Computer supported coop- erative work

Kuzuoka, H., Pitsch, K., Suzuki, Y., Kawaguchi, I., Yamazaki, K., Ya- mazaki, A., Kuno, Y., Luff, P., Heath, C.: Effect of restarts and pauses on achieving a state of mutual orientation between a human and a robot. In: Proceedings of the 2008 ACM conference on Computer supported coop- erative work. pp. 201–204 (2008)

work page 2008