{"id":"0224a2c4-eb9d-4ac6-af5e-c5f7e783128f","arxiv_id":"2412.20826","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ReStory re-arranges frames from an existing HRI video to match the semantic captions of a reference storyboard, producing a new human-interpretable interaction storyboard.","lead":"Researchers built ReStory, a pipeline that uses vision-language models to re-order frames from a different person's robot-interaction video so they match an existing interaction storyboard, creating a new but semantically similar interaction scene. It matters because human-robot interaction datasets are small and expensive to collect, and this offers a way to augment scarce data with human-checkable synthetic scenarios.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Synthesized storyboards may not preserve the target interaction pattern because the robot's tele-operated behavior in the source video is never verified against the base storyboard (assumption in §3.2).","rationale":"The reader's weakest_assumption exactly captures the soft spot I identify: the robot's behavior in the synthesized storyboard is assumed to match the base storyboard without verification, and this assumption is load-bearing because robot motion is part of the interaction signature. The paper is honest about the assumption in §3.2 and lists related limitations in §5.2, but it never tests this particular assumption, and the evaluation design (in-group annotators, privileged dataset access, ego-motion text) cannot disambiguate whether narrations reflect the actual images or the injected robot-motion text. This concern directly targets the central claim that ReStory 'is capable of synthesizing human-interpretable interaction scenarios' that preserve semantic signatures. A concrete trajectory comparison would settle whether the concern lands. If the robot motion mismatches, the validation results are explained by the textual ego-motion cue rather than by the pipeline's frame selection, making the headline claim unsupported. If the robot motion matches (e.g., the tele-operator happened to repeat the same approach in all clips), then the assumption holds in this particular demonstration and the proof-of-concept remains plausible. Because the paper is positioned as a preliminary proof-of-concept with explicit caveats, this concern does not change the reader's CONDITIONAL verdict; it sharpens the reason why conditions are needed.","tokens_in":7373,"tokens_out":4081,"duration_ms":41300,"concrete_test":"For each of the six synthesized storyboards, compute the robot's trajectory from telemetry or visual odometry in the source video between every pair of selected frames, and compare it to the base storyboard's robot action sequence (e.g., driveby: robot passes without stopping; ask-and-receive: robot approaches and waits). Also document whether the ego-motion text provided to annotators was taken from the reference storyboard or from the source video. If the robot's behavior differs from the reference, or if the ego-motion text is from the reference while the frames are from another video, the synthesized storyboards do not depict the claimed interaction signatures and the validation cannot support the central claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that ReStory generates storyboards that 'capture the semantic-level signatures' of reference interaction sequences (§5.1). For sequences like driveby, offer-and-release, and ask-and-receive, the robot's motion is a defining component: driveby requires the robot to pass by; ask-and-receive requires it to approach and wait. The method only matches human pose and interaction captions, not robot behavior. Section 3.2 explicitly states the implicit assumption that 'the behavior of the robot in the new storyboard must match that in the old storyboard,' and in the demonstrated case the robot is tele-operated, so this match is not automatic. The paper provides no verification that the robot's motion in the source video (from which the selected frames are drawn) matches the reference storyboard's robot behavior. In fact, the source videos depict different interaction patterns by design (§4.1), so the robot's approach/stop/pass behavior in those videos is likely different. The evaluation (§4.2) supplements the storyboards with ego-motion text between frames, but the paper does not state whether that text comes from the reference storyboard or the source video. If it comes from the reference, the storyboards combine human frames from one interaction with robot motion from another, creating an internally inconsistent scenario; if it comes from the source, the robot behavior is not the reference behavior and the target signature is not preserved. Either way, the validation result—annotators often recognizing the target pattern—cannot be attributed to the image content, and the central claim that the synthesized storyboards capture the semantic signature is not supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"ReStory proposes a VLM-driven pipeline for augmenting scarce human-robot interaction (HRI) datasets. Given an ethnographer-crafted reference storyboard and another interaction video, the method captions each reference keyframe and each sampled video frame with two prompts (body posture P1 and interaction context P2), computes weighted SBERT semantic similarities between captions, and retrieves for each reference keyframe the video frame with the highest similarity. The resulting storyboards preserve the high-level action sequence of the reference while introducing a different person's visual appearance and idiosyncratic movements. The authors validate the approach by asking seven in-group researchers to narrate three original storyboards and six synthesized storyboards built from cross-pairings of the three interaction patterns (driveby, offer-and-release, ask-and-receive) from Brown et al.'s trash-barrel robot dataset. They report that annotators frequently, but not always, described the defining signatures of the intended patterns, and they conclude that the preliminary validation demonstrates ReStory's ability to generate storyboards that capture the semantic-level signatures of the base sequences. The paper is explicitly positioned as a proof-of-concept requiring human supervision.","tokens_in":7627,"tokens_out":4061,"duration_ms":42479,"significance":"If the central claim were properly established, ReStory would offer a useful, low-cost way to repurpose existing in-the-wild HRI video into new storyboard-style training or design material, and it would connect EMCA storyboard practices with modern VLM capabilities. The paper's strengths are its clear and modular pipeline description, its transparent enumeration of assumptions (especially the robot-behavior assumption in §3.2), and its candid discussion of limitations in §5.2, including distance perception, causality ordering, and VLM hallucination. However, the reported validation is too weak to support the conclusion as stated: it lacks a control condition, uses a small convenience sample without inter-annotator agreement metrics, and does not verify a load-bearing assumption about robot behavior. The contribution at this stage is therefore a promising proof-of-concept with an unproven efficacy claim, rather than a validated data-augmentation method.","major_comments":[{"comment":"The validation study lacks any control condition. Annotators were shown only storyboards produced by the full ReStory pipeline, so the observed interpretability cannot be attributed to the similarity-based retrieval. A natural control would be randomly sampled frames from the same input videos, or a similarity metric based on random captions, matched for frame count and ordering. Without such a baseline, the counts reported in §4.2 are consistent with the trivial explanation that any frames drawn from a coherent interaction video—regardless of their semantic alignment with the reference storyboard—can be narrated by a motivated annotator.","section":"§4.1–§4.2"},{"comment":"The paper explicitly states in §3.2 the implicit assumption that the robot's behavior in the new storyboard must match that in the old storyboard, but it never verifies this assumption. The source videos are chosen because they depict different interaction patterns than the reference (e.g., driveby synthesized from offer-and-release footage), and the robot is tele-operated; nothing in the pipeline checks whether the robot's approach, stopping, and passing motion in the source footage matches the reference pattern. The evaluation in §4.1 supplements the storyboards with ego-motion text between consecutive frames, but the paper does not state where this text comes from—the reference storyboard or the source video. If it comes from the reference, the storyboards combine human frames from one interaction with robot motion from another; if it comes from the source, the robot behavior is not the reference behavior and the defining signature of the target interaction may be lost. Either way, the evaluation does not address this load-bearing assumption.","section":"§3.2 and §4.1"},{"comment":"The evaluation is circular and under-powered. The prompts P1 and P2 are designed to capture exactly the kind of posture and interaction-context information that defines the three interaction categories, and the retrieval objective is to match the reference storyboard's captions; thus, the synthesized storyboards are constructed to align semantically with the reference at the caption level. Asking annotators to narrate the same three categories then partly measures the consistency of the caption-recaption loop, not the validity of the synthesized interaction. Additionally, seven annotators from the same research group, with no reported inter-annotator agreement measure and no statistical comparison against chance, cannot support the quantitative conclusion in §5.1. The reported counts (e.g., 'six out of seven' for original patterns, 'seven out of fourteen' for synthesized driveby) need a defined success threshold, confidence intervals, or a chance-level baseline to be interpretable.","section":"§4.1–§4.2"},{"comment":"The results reported in §4.2 do not support the claim in §5.1 that the preliminary validation 'demonstrates that ReStory is able to generate new storyboards that capture the semantic-level signatures of these sequences.' For the synthesized driveby pattern, only 7 of 14 narrations captured the user's avoidance behavior, and for the synthesized ask-and-receive pattern, only 6 of 14 captured robot initiation—close to what would be expected by chance if annotators narrated arbitrary interactions in this setting. The aggregated counts also pool two synthesized storyboards per pattern, obscuring per-storyboard differences. The paper should report results per storyboard, define a target level of agreement before the study, and explicitly compare synthesized against original storyboards on the same metric before drawing the conclusion in §5.1.","section":"§4.2 and §5.1"}],"minor_comments":[{"comment":"The annotation task is underspecified: 'narrate frame by frame what happened' and 'report where the storyboard breaks down' are two different tasks, and it is unclear whether annotators were instructed to classify the interaction pattern or simply describe events. The protocol should be described in enough detail to be replicable.","section":"§4.1"},{"comment":"The paper states that annotators were seven researchers within the research group and had access to the dataset but were 'not familiar with the footage.' Please specify whether they were blind to the study hypothesis and whether they had seen the original Brown et al. paper or its storyboards, since this could bias narration toward the expected patterns.","section":"§4.1"},{"comment":"The YOLO cropping of the user from input frames is described in §5.2 as part of the pipeline, but the main pipeline description in §3.1 and Figure 1 do not mention this preprocessing step. Clarify whether cropping was applied in the validation study and how it affects the pose and context captions.","section":"§3.1 and §5.2"},{"comment":"The source of the ego-motion text is not specified—whether it came from robot telemetry, VLM inference, or manual annotation by the authors. This should be stated explicitly because it directly affects the interpretation of the storyboards in the evaluation.","section":"§4.1"},{"comment":"The weighting parameter alpha is fixed at 0.2 without any sensitivity analysis. At minimum, the authors should note that the reported results depend on this choice and that a small sensitivity study would strengthen the claim that the method is robust to the similarity weighting.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a well-scoped proof-of-concept, and its honest limitation section is a strength. However, the validation study as designed cannot support the central claim: the absence of a control, the unverified robot-behavior assumption, and the small unblinded annotator pool are load-bearing issues rather than presentation concerns. These can be fixed within the scope of a revised manuscript by adding a random-frame control, verifying or constraining the robot behavior, and reporting per-storyboard results with agreement metrics. I would encourage the editor to invite a revision rather than reject, as the pipeline idea is clear and potentially useful to the HRI community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper is that the core idea is real and worth a look: use a VLM to caption frames, then use SBERT similarity to retrieve frames from a different video that match a reference EMCA storyboard, and reassemble them into a new storyboard. I have not seen that exact combination before, and for a research community starved of interaction data, it is a sensible angle. The paper is also honest about being a proof-of-concept and openly lists its limitations: distance perception, causality violations, VLM hallucination, and the need for human supervision. That transparency earns credit. What the paper does well: the pipeline is described clearly enough to reproduce in broad strokes, the choice of prompts is motivated, and the framing around EMCA storyboards as templates is a genuine contribution. The authors are not over-claiming in the abstract; they say 'has the potential' and 'human-interpretable,' which is appropriate. The soft spots are mostly in the evaluation. There is no control condition, so the fact that annotators could narrate the synthesized storyboards does not tell you ReStory's similarity-based retrieval is doing the work; random frame selection might do just as well. The annotators are seven people from the same research group with dataset access, which biases them toward recognizing the patterns. No code, data, or exact prompts are released, so the results are not independently checkable. The hand-set alpha (0.2) and frame sampling rate have no sensitivity analysis. One storyboard needed manual frame reordering, which the paper reports honestly but which weakens the 'automatic synthesis' story. The stress-test note about robot behavior is the most serious issue, and it holds up. Section 3.2 explicitly assumes the robot's behavior in the new storyboard matches the reference. In the demo, the robot is tele-operated, and the source videos are chosen because they show different interaction patterns. So the robot's approach, stop, and pass trajectories almost certainly differ. The evaluation provides ego-motion text between frames, but it never says whether that text comes from the reference storyboard or the source video. Either way, the synthesized storyboard may be mixing human frames from one interaction with robot motion from another, and the results cannot be attributed to the image content alone. The paper acknowledges the assumption but never tests it, which is a load-bearing gap. Bottom line: this is a promising proof-of-concept for HRI researchers and interaction designers who work with EMCA-style analysis. It deserves a serious referee, but the validation needs a proper baseline, outside annotators, and an explicit test of the robot-motion assumption before the central claim is supported. I would not cite it yet, but I would read a revised version.","headline":"A genuinely novel pipeline for storyboard-based HRI data augmentation, but the validation cannot yet carry the claim that the synthesized storyboards capture the target interaction pattern.","tokens_in":718,"tokens_out":857,"would_cite":false,"duration_ms":23733,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ReStory, a vision-language pipeline, can reorder a second person's actions into the sequence of a reference interaction storyboard, producing new human-interpretable HRI scenarios.","keywords":["Synthetic human-robot interaction","Data augmentation","Vision-language models","Storyboarding","Ethnomethodology","Sentence-BERT","Frame retrieval"],"falsifier":"Use ReStory with a source video in which the robot approaches the person from a different side or at a different speed than in the reference storyboard, then ask annotators to name the interaction pattern; if the generated storyboard no longer conveys the reference pattern, the key assumption that robot behavior is constant is violated.","tokens_in":7136,"feed_emoji":"🤖","tokens_out":9040,"duration_ms":75400,"temperature":0.7,"pith_summary":"Human-robot interaction researchers lack internet-scale datasets, so they cannot easily train data-hungry models. ReStory proposes to augment existing in-the-wild HRI footage by re-splicing one participant's actions into the storyboard of another interaction. The paper claims that, given an ethnographer-crafted storyboard and a second video, a vision-language model can caption both, and a semantic similarity metric can find the video frames that best match each storyboard keyframe, yielding a new storyboard of a different person performing the same interaction pattern. A validation with seven annotators suggests the synthesized storyboards convey the defining semantic signatures of the base patterns, such as avoidance in the driveby scenario. The approach still requires human supervision, but it offers a new way to stretch scarce interaction data.","feed_headline":"ReStory turns one interaction video into many new HRI storyboards","feed_subtitle":"It matches frames by meaning, not pixels, so scarce robot-interaction footage yields new, human-readable scenarios.","key_machinery":"The central machinery is the semantic frame-retrieval loop: a two-prompt VLM captioning scheme (posture first, then interaction context conditioned on posture and previous context) combined with SBERT embedding and a weighted cosine-similarity score (context weight 0.8, pose weight 0.2) that selects the best-matching frame from a source video for each storyboard keyframe. This loop aligns the source person's atomic actions with the reference sequence's semantic content, ignoring pixel-level differences such as left- vs right-handed waving, so that the new storyboard preserves the high-level interaction pattern while introducing fresh visual variation.","core_discovery":"ReStory rests on the idea that an interaction can be compressed into a storyboard: a sequence of keyframes whose captions describe the postures and contextual actions of the people involved. The method takes an existing EMCA storyboard as a reference and a different interaction video as a source. For each storyboard frame and for two-frames-per-second samples of the source video, ReStory prompts a vision-language model to produce a posture caption (P1) and, conditioned on that posture and the previous frame's context, an interaction-context caption (P2). It then computes the weighted cosine similarity between SBERT embeddings of the captions, with context weighted more heavily than posture (α = 0.2), and selects for each storyboard frame the source frame with the highest similarity. The selected frames are stitched, with the robot's ego-motion between consecutive frames given as text, to form a new storyboard in which the source person appears to perform the reference interaction pattern. The paper's claim is that this produces human-interpretable scenarios that preserve the semantic-level signatures of the base sequences, as suggested by the narration study where annotators recognized the defining features of the driveby, offer-and-release, and ask-and-receive patterns at rates comparable to the original storyboards.","pith_inferences":["A natural next step is to apply the same retrieval idea to the robot side, swapping robot behaviors while keeping human footage fixed; this would require solving the paper's weakest assumption about robot behavior being constant and could enable two-party storyboard synthesis.","The effect of the distance confound could be quantified by adding depth as an input feature and measuring whether annotator agreement on the focal action improves; this is a direct testable extension suggested by the paper's own limitation discussion.","ReStory could serve as a data-generation module for training perception models: caption-aligned frame pairs from the same interaction pattern provide weak supervision for pose or action recognition, bypassing the need for manual annotation of new footage.","Because the similarity metric is a weighted combination, the relative importance of posture and context could be tuned per task; for example, interaction phases that hinge on handover might require higher context weight, whereas posture-driven signals like avoidance might need higher pose weight."],"forward_implications":["Existing HRI datasets can be multiplied by combining each interaction pattern with different participants' footage, generating new storyboard episodes without additional real-world collection.","The prompts and similarity metric are user-defined, so the pipeline can be retargeted to other robot form factors or interaction contexts by changing what the VLM is asked to describe and what similarity weights are used.","The synthesized storyboards can support design ideation, letting interaction designers explore how a different user might experience the same robot behavior.","Because frames are drawn from real footage, personal idiosyncrasies (gesture style, gaze, posture) are preserved even as the action sequence is rearranged, yielding a form of data augmentation that respects individual variation.","Human verification remains necessary: the paper reports that causality violations (two frames swapped) and VLM hallucinations can occur, so the output storyboards are semi-novel rather than fully automatic."],"supporting_citations":[{"why":"Supplies the reference interaction patterns (driveby, offer and release, ask and receive) and the original video clips from which the base storyboards are built.","marker":"[3]"},{"why":"Provides the trash barrel robot footage and urban plaza storyboards that the demonstration re-splices.","marker":"[4]"},{"why":"Grounds ReStory in storyboard-based EMCA analysis, establishing the idea that keyframes compress an interaction sequence.","marker":"[16]"},{"why":"Defines the sentence-embedding similarity metric (SBERT) used to match storyboard frames to source video frames.","marker":"[19]"},{"why":"Supplies the object detector that crops the person from frames, reducing VLM hallucination in the pipeline.","marker":"[18]"}],"fun_headline_variants":["ReStory: Match semantics, not pixels, to make storyboards","ReStory: Turn one video into many HRI scenarios","ReStory: VLM-crafted storyboards from a single interaction clip","ReStory: Semantic storyboards from scarce HRI footage","ReStory: One video, many storyboards via semantic matching"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"ReStory assumes that the robot's motion in the generated storyboard matches the robot's motion in the reference storyboard; if the robot approaches or moves differently, the spliced human frames no longer tell the same interaction story.","fun_headline_variants_meta":{"raw":{"variants":["ReStory: Match semantics, not pixels, to make storyboards","ReStory: Turn one video into many HRI scenarios","ReStory: VLM-crafted storyboards from a single interaction clip","ReStory: Semantic storyboards from scarce HRI footage","ReStory: One video, many storyboards via semantic matching"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000457,"raw_usage":{"total_tokens":2282,"prompt_tokens":926,"completion_tokens":1356,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":1269}},"tokens_in":542,"tokens_out":1356,"duration_ms":10988,"temperature":1.0,"reasoning_tokens":1269,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:09:44.858010+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use ReStory with a source video in which the robot approaches the person from a different side or at a different speed than in the reference storyboard, then ask annotators to name the interaction pattern; if the generated storyboard no longer conveys the reference pattern, the key assumption that robot behavior is constant is violated.","supporting_citations":[{"cited_title":"In: Companion of the 2023 ACM/IEEE International Conference on Human- Robot Interaction","cited_arxiv_id":null,"evidence_quote":"Provides the trash barrel robot footage and urban plaza storyboards that the demonstration re-splices."},{"cited_title":"In: Proceedings of the 2024 ACM/IEEE Inter- national Conference on Human-Robot Interaction","cited_arxiv_id":null,"evidence_quote":"Grounds ReStory in storyboard-based EMCA analysis, establishing the idea that keyframes compress an interaction sequence."}],"review_version":1}