{"id":"1878a98d-0427-4918-8520-2a1e5831e983","arxiv_id":"2504.13351","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CoM sequentially prompts a VLM over video, force/audio, and hand pose, yielding roughly threefold better extraction of task plans and control parameters, with real robots succeeding in 73% of trials.","lead":"This paper introduces Chain-of-Modality, a prompting strategy in which a vision-language model analyzes each signal in a human demonstration video sequentially, then turns the combined analysis into a robot control program. The authors report that this step-by-step approach extracts task plans and force parameters far more reliably than sending all modalities at once, and that the resulting programs work on real robots across several manipulation tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CoM's claimed gains rest on unvalidated scalar summaries of EMG/audio and 2D fingertip tracks; if these carry artifacts, the reported advantage may not transfer to new recordings.","rationale":"The reader's weakest assumption identifies the same load-bearing premise: the derived force and hand-pose signals must carry enough reliable information for the VLM to segment tasks and assign control parameters. I agree, and I add that the paper gives no direct evidence for this premise, only the indirect evidence of the benchmark results. That is not fatal to the core comparison on the current dataset, because the force ablations in Table I do show that removing force lowers performance, but the generalization claim extends beyond the current recordings and subjects. My proposed shuffle test would distinguish between the force signal actually being used versus the VLM relying on visual context, task priors, or prompt artifacts. Since the reader's CONDITIONAL verdict already conditions on reliability and reproducibility of the empirical claim, my concern does not change the verdict.","tokens_in":12647,"tokens_out":9414,"duration_ms":101241,"concrete_test":"Take the Opening Bottle and Inserting Plug test videos and run CoM with the force trace replaced by a random time-shuffle of itself, keeping images, hand pose, and all prompts fixed. If exact-plan accuracy does not drop substantially toward the vision-only baseline, the force channel is not doing the causal work and CoM's advantage cannot be attributed to the multimodal signal; if it does drop, the concern is settled in favor of the paper. In the same harness, compare max-pooled 8-channel EMG against a single representative channel to test whether the max-pooling choice itself is robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III.D defines the force signal as the per-timestep maximum across eight EMG channels (after downsampling to 60 Hz) and the audio signal as loudness; hand pose is the 2D pixel locations of fingertips from HaMeR. CoM's segmentation and control-parameter inference (e.g., 'twist counterclockwise 180°', force=20 vs 100 in the Section IV.B examples) must be recoverable from these summaries. The paper provides no ablation or calibration showing that the max-pooled envelope reliably tracks grasp/release timing and relative effort across subjects, sessions, or armband placements, and no failure analysis of HaMeR 2D tracks under occlusion or viewpoint change. If those traces contain artifacts, such as co-activation from adjacent muscles, background audio, or projection-dependent fingertip motion, CoM's per-modality analyses inherit the errors, and the high accuracy on the 10-video benchmark need not transfer to new recordings. This is load-bearing because the core mechanism is progressive integration of these signals; if the scalar force trace does not reliably mark action boundaries and relative effort, the segmentation and control-parameter assignment have no independent ground truth to latch onto. The paper's own limitation that audio captures only volume extends to the force channel, but the force preprocessing choice is not similarly flagged.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Chain-of-Modality (CoM), a prompting strategy that feeds a multimodal human demonstration video (RGB frames, a scalar force trace derived from EMG or audio, and 2D hand-pose keypoints) to a VLM one modality at a time, progressively refining a task plan and control parameters. A second VLM pass converts the resulting analysis into Python code that calls a predefined robot skill library. The experiments compare CoM with modality-ablation and integration-order baselines on four tasks using Gemini 1.5 Pro and GPT-4o, and evaluate the generated programs on real robots for opening bottles, inserting a plug, wiping a board, and playing a drum, reporting an average robot success rate of 73% against a 92% manually coded oracle.","tokens_in":12890,"tokens_out":8491,"duration_ms":79014,"significance":"If the reported results are reproducible, CoM would be a useful, simple recipe: current VLMs handle interleaved multimodal streams poorly, and sequential per-modality refinement materially improves structured extraction from videos, while the EMG/audio-derived force channel is a sensible way to capture control parameters that images alone cannot provide. The paper has genuine strengths: it tests two modern VLMs, evaluates on two robot embodiments, includes an oracle upper bound, uses ground-truth task plans rather than circular self-evaluation, and makes falsifiable quantitative claims. The main limitation of the current contribution is evidentiary: the central comparisons rest on 10 self-collected videos per task with 3 stochastic queries, no error bars or significance tests, an underspecified exact-match metric, and no robot-level baseline comparison. The prompting idea is plausible and not internally inconsistent, but the evidence base is too thin for the strength of the headline claims.","major_comments":[{"comment":"The headline numbers in the Introduction are not traceable to the reported tables. The Introduction states that CoM achieves 60% accuracy and that naive all-modality batch querying averages 17%, and §IV.B states that CoM outperforms Sep-Sep by more than 19% with Gemini and 17% with GPT-4o. However, Table I contains only Image-only, w.o. img, w.o. force, w.o. hand, and All; the CoM, Merg, Merg-Sep, Sep-Merg, and Sep-Sep rows are not tabulated, and the quantitative source is only the referenced Fig. 5. In addition, the Table I 'All' row averages about 59% for Gemini and 29% for GPT-4o, so it is unclear how the 17% figure relates to the 'All' baseline. The central empirical claim needs a complete table with per-method, per-task, per-model accuracy and similarity values so that the 60%, 17%, and 19%/17% improvements can be verified.","section":"Introduction, §IV.A–IV.B"},{"comment":"The evaluation metric is underspecified. The text says the paper reports the 'average success rate in generating the correct human task plans observed in the video, along with the similarity score, which is calculated by finding the longest common string between the output and the ground truth.' It is not stated whether 'accuracy' means exact string equality, whether it is computed per action, per video, or per query, how the three queries per video are aggregated (any correct, majority vote, or mean), or how the ground-truth task plans were obtained and annotated. The phrase 'longest common string' is also ambiguous between longest common substring and longest common subsequence. These choices materially affect Table I and the claimed 60%; the authors should provide the exact evaluation protocol, ideally with the evaluation code.","section":"§IV.A"},{"comment":"The scalar force/audio summaries on which the whole method depends are not validated. Section III.D defines the force signal as the per-timestep maximum across eight EMG channels sampled at 200 Hz and downsampled to 60 Hz, and audio as loudness; Sections III.A–III.B then use the timing and level of force as key cues for segmentation and for control parameters such as twist angle and force intensity. The paper provides no evidence that the max-pooled EMG envelope reliably marks grasp/release events and relative effort, no per-subject or per-session analysis, and no failure analysis for HaMeR 2D fingertip tracks under occlusion or viewpoint change. If muscle co-activation, background audio, or projection-dependent finger motion corrupts these traces, CoM's chain will inherit the error and the 10-video benchmark results may not transfer to new recordings. The conclusion lists only the audio-volume limitation, not the force-envelope preprocessing; please add a calibration or control experiment that aligns envelope peaks with independently annotated contact/release events and reports the force-only stage's accuracy.","section":"§III.A, §III.D, Conclusion"},{"comment":"The on-robot evaluation does not test CoM's central advantage. Table II reports only 'Ours' and an oracle program; no baseline-generated code (for example, from Image-only or Merged analysis) is executed on the robot. The 73% average success rate therefore establishes absolute feasibility, not that CoM's prompting advantage survives to execution, and the claim of 'strong generalization' is not a comparative claim on the current data. The authors should either add robot-level baseline comparisons in the same 20-trial generalization settings, or explicitly restrict the generalization claim to absolute performance.","section":"§IV.B, Table II"},{"comment":"The statistical basis is too thin for the strength of the claims. Each task uses 10 self-collected videos and 3 queries per method, and Table I contains no confidence intervals, no per-video breakdown, and no significance tests. With n=10 and three stochastic samples, differences such as GPT-4o's 0.00 versus 0.40 on Pressing Cube are not established as beyond sampling noise. Report per-video results with confidence intervals or significance tests, or temper the quantitative claims accordingly.","section":"§IV.A, Table I"}],"minor_comments":[{"comment":"Figure references are inconsistent: the text in §IV.B refers to 'Fig. 5, first column' for qualitative Opening Bottle results, while Fig. 4 is the qualitative figure and Fig. 5 is the quantitative comparison; please renumber or fix the references.","section":"§IV.B, Figs. 4–5"},{"comment":"The hand-pose input is described inconsistently: §III.D says HaMeR localizes 'the pixel locations of the fingertips', while §IV.A says the hand pose data 'consists of 2D pixel locations of the thumb and middle fingertips'; specify which fingers are actually provided to the VLM.","section":"§III.D, §IV.A"},{"comment":"The sentence 'This video demonstrates only the key features of primitive skills, which does not include the testing objects or task plans' appears twice, once in the Prompts paragraph and once in the Results paragraph; remove the duplication.","section":"§IV.A–IV.B"},{"comment":"The caption says 'Accuracy / Similarity Score'; please add a legend stating which column is which and include the number of videos and queries per method in the caption.","section":"Table I"},{"comment":"Minor writing issues: 'w.o. img' and 'w.o. image' are used interchangeably, 'as Fig. 3(c) illustrated' should be 'as illustrated in Fig. 3(c)', and 'Gemini 1.5 pro' should be 'Gemini 1.5 Pro' for consistency.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the paper's central empirical claim is stated as precise percentages in the Introduction, but those numbers cannot be reconstructed from the submitted tables and figures, and the robot evaluation does not include any baseline. I would make the complete results table, the metric definition/evaluation code, and a robot-level baseline condition of acceptance. The paper is close to a strong workshop contribution; for a journal-level claim, the statistical and comparative gaps need to be addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is a nicely scoped proof-of-concept that sequential per-modality prompting beats dumping all modalities into one VLM query, and that cheap scalar force/audio summaries help segment manipulation videos. The empirical base is thin, so the strong generalization claims should be read as provisional.\n\nWhat's genuinely new: the CoM prompting strategy applied to human manipulation videos with force/audio, then compiled into robot code. Each piece exists (VLM prompting, HaMeR hand pose, code-as-policies), but the combination is new, and the within-paper ablations make sense. The example prompt is explicitly task-irrelevant, ground truth is external, and no parameters are fitted to the reported accuracies, so there is no circularity red flag. The real-robot numbers (73% success with an oracle at 92%) give useful context, and the authors are upfront about open-loop execution and audio-only-volume limitations.\n\nSoft spots: the evaluation is small. Ten self-collected videos per task, three queries each, no error bars or significance tests. Exact-match accuracy is not precisely defined, and the LCS similarity metric is more forgiving. Robot trials are 20 per task, again without variance reporting. That's typical for a robotics systems paper, but it caps how much weight the numbers can carry.\n\nThe stress-test concern about scalar summaries — max-pooling eight EMG channels and audio loudness — is fair but not fatal. The authors don't calibrate the force trace across subjects, sessions, or armband placements, and they don't analyze HaMeR failures under occlusion. But the ablations show that removing force sharply reduces accuracy, which is indirect evidence that the trace carries real signal. If those traces become artifact-heavy in new recordings, the pipeline's edge could shrink; that's a transfer risk, not a reason to doubt the within-paper comparison.\n\nWho this is for: people working on learning-from-video for robots, and anyone exploring VLM prompting for multimodal temporal reasoning. It's a systems paper with a clear empirical lesson: progressive modality integration helps current VLMs.\n\nRecommendation: send it to peer review. It's not a breakthrough, but it's a sound proof-of-concept with honest limitations and a reproducible artifact link. The referee should push for a sharper definition of exact-match accuracy, per-trial results, and ideally a small cross-subject/session validation of the force signal.","headline":"A sensible step-by-step prompting trick beats single-batch multimodal prompting for extracting manipulation plans from human video, with a plausible but thinly evidenced empirical case.","tokens_in":13414,"tokens_out":3203,"would_cite":true,"duration_ms":32042,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Chain-of-Modality: reading video, force, and hand pose one at a time yields one-shot robot manipulation programs.","keywords":["chain-of-modality","multimodal human video","vision-language models","robot manipulation","one-shot imitation","EMG force signal","task plan extraction","hand pose estimation"],"falsifier":"Run the force-analysis stage on new recordings where ground-truth force is measured independently with a calibrated sensor, and compare the model's reported force-onset times, force-offset times, and discrete force levels to the sensor trace. If the force stage cannot locate the same intervals and levels that the sensor records within a few frames, then the claim that the max-pooled EMG or loudness signal is the decisive modality for task segmentation and control parameters is not supported.","tokens_in":12470,"feed_emoji":"🤖","tokens_out":7245,"duration_ms":64782,"temperature":0.7,"pith_summary":"Chain-of-Modality is a prompting strategy that asks a vision-language model to read a multimodal human demonstration one modality at a time—force or audio first, then hand pose, then video frames—and to refine a timed, parameterized action plan at each step. The paper's central assertion is that this staged reading lets a single human video teach a robot a manipulation program, including control parameters such as force level and twist direction, which ordinary video cannot convey. Across four tasks the staged prompting reached roughly 60% accuracy in extracting the exact task plan and control parameters, while vision-only prompting scored zero and a single merged multimodal query averaged about 17%. On a real robot, the programs generated from the analysis succeeded in about 73% of trials, including on unseen objects and across two robot platforms. A sympathetic reader should take the contribution as an empirical demonstration that prompting structure, not just model scale, determines whether current vision-language models can use non-visual signals.","feed_headline":"One video plus a force trace writes robot code","feed_subtitle":"Sequential reading of EMG, hand pose, and frames lifts plan accuracy from 17% to 60%.","key_machinery":"The load-bearing object is the Chain-of-Modality prompt chain: a fixed sequence of vision-language-model queries in which the answer produced from the force/audio stage is concatenated into the hand-pose query, and that combined answer is concatenated into the image query, followed by a code-generation query. The force modality is a single normalized scalar per frame, obtained by taking the maximum over eight EMG channels or the loudness of an audio recording; the hand modality is the 2D pixel locations of the fingertips from a hand-reconstruction model. What this machinery does is force the model to commit to an intermediate reading of each modality before seeing the next, so the final plan is built by refinement rather than by one-shot cross-modal correlation. It also keeps the answer grounded in the literal timing of the force trace, which is where the paper says task segmentation and control parameters come from.","core_discovery":"The discovery the paper is trying to establish is that a vision-language model can perform one-shot imitation of manipulation from a single multimodal human video if the query is decomposed into a chain of per-modality analyses. In the first stage the model reads only the scalar force trace—max-pooled EMG channels or audio loudness—and reports when and how much force is applied. In the second stage it adds the 2D pixel positions of the fingertips, which lets it infer grasps and rotations; in the third stage it adds the RGB frames, which lets it name objects and assemble the full timed action sequence. The same model then compiles that sequence into Python calls against a fixed robot API. The paper reports exact task-plan accuracy of 60% for this chain, compared with zero for image-only input and 17% for a single merged multimodal query, and an average 73% real-robot success rate across opening bottles, inserting plugs, wiping a board, and playing a drum.","pith_inferences":["Beyond the paper, the same chain idea could apply to other multimodal reasoning tasks where one modality is weak but informative, such as adding thermal or depth channels to video for inspection workflows, so the prompting pattern is transferable.","The paper reduces each EMG burst to a single max value; an editor's test would be whether feeding the per-channel waveform or its spectrogram, instead of the max-pooled scalar, sharpens the model's distinction between force levels beyond the coarse high/medium/low used here.","Because robot execution is open-loop, a natural extension the paper leaves implicit is a closed-loop variant that re-runs the analysis when a skill fails, which would test whether the extracted plan or the execution layer is the actual failure point.","The zero vision-only accuracy suggests a supervised vision-only force regressor might close part of the gap; the paper does not compare against such a learned baseline, so that comparison remains open."],"forward_implications":["The paper's results imply that non-visual channels—muscle activity or impact sound—can supply the control-parameter information that RGB video alone hides, so multimodal capture should be part of any human-video imitation setup for contact-rich tasks.","Because each modality is analyzed before the next is added, the gains do not depend on the vision-language model spontaneously aligning long interleaved sequences; the same staged prompt may continue to help as long-context models improve.","Generated programs target a platform-agnostic Python API, so a single demonstration video can be re-targeted to different robot arms without retraining; the paper demonstrates this on two bi-manual platforms.","The success of the open-vocabulary object-finding API suggests that the bottleneck for video-to-robot transfer is less about low-level trajectory cloning than about extracting the right high-level actions and parameters."],"supporting_citations":[{"why":"One of the two vision-language-model backends used in all video-analysis and code-generation experiments.","marker":"[1]"},{"why":"Long-context vision-language-model work cited as the reason the models can take video and long numerical sequences as inputs.","marker":"[10]"},{"why":"Supplies the approach of generating robot-executable Python API calls from language-model reasoning, used for the final program generation.","marker":"[30]"},{"why":"Provides the 2D fingertip pixel locations used as the hand-pose input modality.","marker":"[38]"},{"why":"The other vision-language-model backend, also used to implement the open-vocabulary object-finding API.","marker":"[40]"}],"fun_headline_variants":["Chain-of-Modality: read force trace, hand pose, then frames","Sequential modality prompts lift robot plan accuracy to 60%","One video plus muscle signals writes robot programs","Force trace plus video: a recipe for robot imitation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a single scalar force trace (max over eight EMG channels or audio loudness) together with 2D fingertip pixel positions carries enough reliable information for the vision-language model to segment the task and set control parameters; if those derived signals are noisy, ambiguous, or poorly synchronized with the video, the staged analysis inherits the error and the reported accuracy gains would not transfer to new recordings.","fun_headline_variants_meta":{"raw":{"variants":["Chain-of-Modality: read force trace, hand pose, then frames","Sequential modality prompts lift robot plan accuracy to 60%","One video plus muscle signals writes robot programs","Force trace plus video: a recipe for robot imitation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000502,"raw_usage":{"total_tokens":2451,"prompt_tokens":939,"completion_tokens":1512,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":1444}},"tokens_in":555,"tokens_out":1512,"duration_ms":12408,"temperature":1.0,"reasoning_tokens":1444,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:10:03.861688+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the force-analysis stage on new recordings where ground-truth force is measured independently with a calibrated sensor, and compare the model's reported force-onset times, force-offset times, and discrete force levels to the sensor trace. If the force stage cannot locate the same intervals and levels that the sensor records within a few frames, then the claim that the max-pooled EMG or loudness signal is the decisive modality for task segmentation and control parameters is not supported.","supporting_citations":[{"cited_title":"Code as policies: Language model programs for embodied control,","cited_arxiv_id":null,"evidence_quote":"Supplies the approach of generating robot-executable Python API calls from language-model reasoning, used for the final program generation."},{"cited_title":"Reconstructing hands in 3D with transformers,","cited_arxiv_id":null,"evidence_quote":"Provides the 2D fingertip pixel locations used as the hand-pose input modality."}],"review_version":1}