{"id":"7d2440b5-1927-4bab-ae8b-54f602131db2","arxiv_id":"2411.15711","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A vision-plus-speech, hierarchy-based robot assistant reduces assembly time by about 16% and reaches a 92% success rate in a 10-person long-term toy-car co-assembly study, though the reported gains lack statistical tests.","lead":"This paper presents a robot system that watches a person and listens to their voice commands while they assemble a toy car together, and uses a hand-built task map to predict what the person will do next. In tests with 10 people, the full system finished tasks about 16% faster and succeeded 92% of the time, but the statistics are rough and the headline numbers do not always match the tables.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline improvements misreport absolute differences as percentages: the 15.85 percentage-point time gap is called a 15.85% reduction (actual ~21.6%), and the 0.2404 MSE drop is called 24% (actual 19.1%). The central quantitative claim needs recalculation and significance testing.","rationale":"The reader's stated weakest assumption is the completeness of the manually constructed task graph and the reliability of DTW alignment (Eqs. 4-5). I agree that this is a genuine limitation, especially because the paper itself acknowledges in Sec. VI that the task graph requires manual design for specific scenarios. However, the most load-bearing threat to the paper's central quantitative claim is more immediate: the two headline improvement numbers (15.85% time reduction, 24% MSE reduction) do not match the tables. The 15.85 figure is precisely the percentage-point gap in normalized completion times, and the 24 figure is the raw MSE difference; each should have been expressed as a relative percentage. This is not a stylistic quibble: if the numbers are corrected, the relative time reduction is ~21.6% and the MSE reduction is ~19.1%, and the abstract's '15.9%' is wrong. Moreover, the absence of significance tests means we cannot distinguish the reported success-rate differences from sampling noise; some Task-2 comparisons are based on 40 trials per cell with overlapping confidence intervals. These issues do not warrant rejection: the system is deployed on a real KINOVA robot, the study is within-subject with 720 trials, code is linked, and the direction of the effects is consistently in favor of the full framework. The correct disposition remains conditional on a corrected analysis and significance testing, matching the reader's verdict. I therefore recommend no change to the verdict, while noting that the weakest assumption named by the reader is a different (though real) concern.","tokens_in":14734,"tokens_out":10981,"duration_ms":92274,"concrete_test":"Recompute the six group averages from Tables II and IV using the comparison rule stated in Sec. V-C3: average the (PP=0,PM=0) and (PP=0,PM=1) rows as the baseline and compute (baseline - best)/baseline. If the result is ~21.6% rather than 15.85%, correct the abstract, Sec. V-C3, and any derived claims; also relabel the Table V MSE change as a 19.1% relative reduction. Then, using per-participant trial data (10 users, 4 repetitions per condition), compute paired 95% confidence intervals or a permutation test for the success-rate and completion-time differences between (PP=1,PM=2) and the stated baseline, with multiple-comparison correction across the six groups. If the differences are not significant, soften the central claim to 'observed improvements in this study' rather than 'reductions/increases'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the full framework (PP=1, PM=2) reduces task completion time by 15.85% is not supported by the reported table under the comparison stated in Sec. V-C3. The text compares against 'groups lacking both plan prediction and multimodal perception (PP=0, PM=0&1)'. From Table IV, the (PP=0,PM=0) average is (78.80+86.80+53.19)/3 = 72.93 and the (PP=0,PM=1) average is (80.05+79.34+62.33)/3 = 73.91; the combined baseline mean is 73.42. The best group averages (67.42+65.83+39.45)/3 = 57.57. The relative reduction is (73.42-57.57)/73.42 = 21.6%, not 15.85%. The 15.85 figure is exactly the absolute percentage-point gap (73.42-57.57), so the paper confuses percentage points with relative percent reduction. The same pattern appears in Sec. V-C4: Table V shows pose MSE dropping from 1.2574 to 1.017, a relative reduction of 19.1%, yet the text reports a '24% reduction'—again the absolute difference (0.2404) rounded to 24%. No significance tests or confidence intervals are reported for any user-study comparison, and several pairwise claims (e.g., 40-trial Task-2 success differences) would be marginal even before multiple-comparison correction. Consequently, the magnitude and robustness of the headline improvements are not verifiable from the paper, although the direction of the effects is plausible.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multimodal and hierarchical framework for long-term human-robot collaboration. The task is modeled as a hierarchical task graph known to both human and robot; the framework integrates RGB-D vision with speech commands, a hierarchical pose detection module, a hierarchical plan predictor based on DTW alignment to reference sequences, and an online adaptation mechanism for physical action prediction. The system is deployed on a KINOVA Gen3 arm in a toy-car assembly task, and a user study with 10 participants, 6 conditions, and 720 trials is used to evaluate success rate, completion time, adaptability, and user satisfaction. The authors report that the full system achieves a 91.79% average success rate, reduces task completion time by 15.9%, improves pose MSE by 24%, and yields higher user satisfaction, concluding that the framework robustly improves long-term HRC.","tokens_in":15067,"tokens_out":8114,"duration_ms":65260,"significance":"If the reported results are correct, this work is a useful integration of multimodal perception, hierarchical plan prediction, and online adaptation in a physically deployed long-horizon assembly task. The authors provide a real-robot user study with an ablation structure across perception modality and plan prediction, as well as open-source code and a demo video, which are valuable for reproducibility. However, the quantitative evidence is undermined by a systematic confusion between absolute percentage-point differences and relative percentage improvements, and by the complete absence of significance tests or confidence intervals. Because the headline numbers in the abstract and conclusion rest on these statistics, the current form does not support the strength of the claims. The central direction of the effect—full system outperforming ablations—is plausible, but the magnitude and robustness need re-analysis.","major_comments":[{"comment":"The reported improvements are absolute differences presented as relative percentages. In Sec. V-C3, the text states an 'average reduction of 15.85%' for the best group (PP=1, PM=2) versus groups lacking both components (PP=0, PM=0&1). Using Table IV, the baseline mean is (78.80+86.80+53.19)/3 for PP=0/PM=0 and (80.05+79.34+62.33)/3 for PP=0/PM=1, giving a pooled baseline of 73.42; the best group mean is (67.42+65.83+39.45)/3 = 57.57. The relative reduction is (73.42-57.57)/73.42 = 21.6%, not 15.85%, which is exactly the absolute percentage-point gap. Similarly, in Sec. V-C4, the pose MSE reduction from 1.2574 to 1.017 is a relative decrease of 19.1%, not the reported 24%; the absolute difference 0.2404 was rounded to 24%. The '14.22%' success-rate increase in Sec. V-C1 for PP=1/PM=2 versus PP=0/PM=2 is also an absolute difference of 14.2 percentage points, not a relative improvement, which would be about 18.3%. All such comparisons must be recomputed and explicitly labeled as relative percentages or percentage points.","section":"Sec. V-C3, V-C4 and V-C1"},{"comment":"The manuscript repeatedly claims 'significant' improvements (e.g., 'significantly reduces task completion time', 'significantly improve action prediction accuracy'), but no significance tests, confidence intervals, or effect sizes are reported for any user-study comparison. Tables II and IV provide means and standard deviations over 40 trials per cell, yet these trials are nested within 10 participants and are not independent; an appropriate repeated-measures or mixed-effects analysis is required. Many pairwise differences are small relative to the reported standard deviations (for example, Task 1 success rates: 74.45±17.15 vs 70.63±18.65), so without statistical evidence the claim that the integrated framework robustly improves long-term HRC is not verifiable. In addition, the multiple comparisons across six groups and three tasks need correction for multiplicity.","section":"Sec. V-C1 through V-C5"},{"comment":"The user study instructs participants to perform each task 'using any plan' and does not report the distribution of plans chosen in each condition. Because the framework's success depends on predicting plans over the manually designed task graph, systematic differences in plan selection across conditions (for instance, if participants in the full-system condition happened to choose easier plans) would confound all group comparisons. The paper should report plan frequencies per condition and, ideally, analyze outcomes per plan or include plan as a covariate in the statistical model.","section":"Sec. V-B3"},{"comment":"The central novelty of the paper is hierarchical plan prediction (Eq. 4-5), but no direct metric for plan prediction accuracy is reported. The offline evaluation in Sec. V-C4 measures physical action prediction (intention accuracy and pose MSE), and the user study reports only downstream task success, completion time, and satisfaction. Because the plan prediction module is the key contribution, an evaluation of how often the predicted reference sequence R* matches the human's actual plan, and of how DTW alignment errors affect downstream performance, is necessary to support the claim that hierarchical plan prediction significantly enhances the system.","section":"Sec. IV-E and Sec. V-C"}],"minor_comments":[{"comment":"The proof of the mutual-information inequality contains an algebraic error: H(g,OV)-H(g,OV,OA)+H(OA|OV) is not equal to H(g,OV|OA)+H(OA|OV). The inequality itself is true because conditioning on additional variables cannot increase entropy, but the derivation should be corrected.","section":"Appendix A"},{"comment":"Typo: 'multual information' should be 'mutual information'.","section":"Sec. IV-A"},{"comment":"The text reports a variance of 0.013 for the Kalman-filtered naive detection but does not report the corresponding variance for the proposed hierarchical method; please add a numerical comparison and error bars for all methods.","section":"Fig. 7"},{"comment":"The claims that plan prediction improves user plan satisfaction by 47.5% and multimodal perception by 27.9% should clarify whether these are relative increases or percentage-point differences; the same absolute-vs-relative ambiguity as in Sec. V-C3 appears here.","section":"Sec. V-C5"},{"comment":"For clarity, specify the order of sessions and whether participants had prior exposure to the full system; also state how the 'any plan' choices were elicited and recorded.","section":"Sec. V-B3"},{"comment":"Typos: 'task progree' should be 'task progress'; 'in future research' should be 'in future work'.","section":"Sec. VI"}],"recommendation":"major_revision","confidential_remarks":"The systematic confusion between percentage points and relative percentages in the headline numbers is concerning, but it is fixable by recalculation and explicit wording. The more substantial gap is the absence of any statistical inference for the user study; for a paper that repeatedly claims 'significant improvements', this must be addressed. The evaluation is also limited to a single toy-car assembly domain with a manually designed task graph, so the generalizability claims in the title and abstract should be tempered. I would support reconsideration after a thorough revision that corrects the statistics and adds appropriate significance testing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a system paper, not a theory paper, and judged on those terms it mostly delivers. The genuinely new thing is the integration: vision plus speech, two-level plan prediction on a hierarchical task graph, online adaptation, all deployed on a Kinova arm in a long-horizon assembly task, with code released. The ablation across six conditions (PP x PM) is a reasonable design, and the user study with 720 trials gives the central claim—the full framework helps—a fair amount of empirical support. The self-citations to your prior plan-recognition and adaptation work are legitimate building blocks, not circular reasoning.\n\nThe soft spots are real and need airing. The stress-test note is right: the 15.85% completion-time reduction is actually a percentage-point gap, not a relative reduction. From Table IV the relative reduction is about 21.6%. Likewise the 24% MSE improvement is really 19.1% (0.2404/1.2574). These are headline numbers in the abstract, so the paper is misreporting its own results. That has to be fixed, and the same pass should add significance tests or confidence intervals—with 10 participants and the large standard deviations in Tables II and IV, claims like the 14.22% success-rate improvement need statistical support. The appendix proof of the mutual-information inequality is also wrong: the algebraic step relating H(g|OV) - H(g|OV,OA) to H(g,OV|OA) + H(OA|OV) does not hold. The inequality itself is true, so this is a fixable but genuine flaw.\n\nThe hand-designed task graph and the fixed-object-position assumption are acknowledged limitations, and they correctly bound the scope: this is a demonstration on one toy-car task, not a general framework. That is fine, as long as the claims stay calibrated to that scope. The paper is honest about these limits, which counts in its favor.\n\nWho gets value from this: researchers building real HRC systems for assembly, especially those working on multimodal fusion or proactive robot assistance. It deserves a serious referee, but the referee should demand corrected numbers, significance testing, and a corrected appendix proof. I would not desk-reject this; I would send it back for major revision with the expectation that a cleaned-up version is publishable.","headline":"A useful system-integration paper with a real robot study, but the headline numbers are misreported and the statistics need work before this is publishable as-is.","tokens_in":649,"tokens_out":660,"would_cite":true,"duration_ms":26580,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that integrating vision and speech with hierarchical plan prediction over a shared task graph makes long-term human-robot collaboration more successful, faster, and more satisfying.","keywords":["long-term human-robot collaboration","multimodal perception","hierarchical plan prediction","task graph","human pose detection","speech recognition","assembly task","online adaptation"],"falsifier":"Run the same toy-car assembly with a participant instructed to follow an assembly order that is deliberately absent from the task graph, or place the tubes in a new location, and check whether plan predictions and task success stay at the reported 91.79% level; a significant drop would confirm that the manual-graph and fixed-object assumptions carry the stated result.","tokens_in":14490,"feed_emoji":"🤖","tokens_out":7568,"duration_ms":65352,"temperature":0.7,"pith_summary":"This paper is trying to establish that long-term human-robot collaboration can be made substantially more reliable by combining multimodal perception (video plus speech) with hierarchical plan prediction over a structured task graph. The authors report that this integrated framework reaches a 91.79% task success rate in a real-world toy-car assembly user study, cuts average completion time by 15.85% relative to ablation groups without both components, and earns an overall user satisfaction score of 84%. A sympathetic reader should care because long-horizon collaboration is where robots currently fail: uncertainty compounds over many subtasks, and the paper's claim is that two-level plan prediction plus speech correction prevents those errors from cascading.","feed_headline":"Robot teamwork framework hits 91.8% success in long assembly tasks","feed_subtitle":"Vision plus speech and hierarchical plan prediction cut completion time by 15.9% in user studies.","key_machinery":"The central object is the hierarchical task graph: a directed And-Or graph whose nodes are subtasks and whose edges encode which orders are valid, with each subtask represented as a small temporal plan graph of human and robot actions. The load-bearing mechanism is the plan-prediction module, which uses dynamic time warping to align the recorded task progress to the closest reference node sequence in that graph (Eq. 4) and then maps the current detected action onto the most likely next action within that sequence (Eq. 5). Around this sit the supporting mechanisms: multimodal fusion of visual pose and speech commands, hierarchical pose detection that crops to the nearest human, sensitivity-aware action classification that converts dangerous misclassification errors into safer ones, and an online adaptation rule that tunes the trajectory predictor to each user.","core_discovery":"The paper's central claim is that robustness in long-term human-robot collaboration comes from structuring the whole task as a hierarchical task graph and giving the robot two complementary ways to stay aligned with the human: multimodal perception that lets a spoken command confirm or override a visual interpretation, and hierarchical plan prediction that first matches observed task progress to a reference node sequence in the graph using dynamic time warping (Eq. 4) and then refines the predicted action to the most probable next action in that sequence (Eq. 5). In the physical assembly experiments, the full system outperforms every ablation on success rate, completion time, and user satisfaction; the vision-only condition degrades sharply when users may choose among plans, and adding hierarchical plan prediction to multimodal perception raises success by 14.22%. The authors also show that hierarchical pose detection, which focuses on the closest human and ignores others, lowers keypoint deviation in multi-human scenes and raises action-prediction accuracy from about 51% without hierarchy to about 96% with it.","pith_inferences":["Because the task graph is hand-built per scenario, the framework's scalability bottleneck is graph construction; automating that construction from demonstrations or natural-language instructions is the natural next step.","The DTW-based plan alignment should transfer to any long-horizon collaborative domain (cooking, kitting, maintenance) where a task graph exists, which means the method's generality is bounded by graph availability rather than by assembly specifically.","The fixed-object assumption means the robustness claim has not yet been tested against clutter or displacement; an object-tracking extension would be the direct stress test of the framework's real-world applicability."],"forward_implications":["Speech commands give the robot a way to recover from a wrong visual guess mid-task, which is what keeps user plan satisfaction high when plans are flexible.","Proactive, prediction-driven assistance translates directly into time savings, with the full system completing tasks on average 15.85% faster than groups lacking both components.","Hierarchical pose detection by proximity is what makes action prediction accurate in multi-human scenes, jumping from roughly 51% to 96% accuracy.","Online adaptation lowers trajectory-prediction error by about 24% for new users, so the framework does not require retraining per person.","Users report that the system lets them focus on the assembly task itself and makes them more willing to keep collaborating for extended periods."],"supporting_citations":[{"why":"Supplies the And-Or graph formalism used to structure the long-term task into alternative plan paths.","marker":"[15]"},{"why":"Provides the dynamic time warping algorithm used to match observed task progress to reference plan sequences.","marker":"[31]"},{"why":"Supplies the online model adaptation method used to tune trajectory and action prediction to each user.","marker":"[3]"},{"why":"Supplies the DLinear time-series architecture used for trajectory and physical action prediction.","marker":"[35]"},{"why":"Provides the pretrained pose estimator from which human keypoints are extracted.","marker":"[8]"},{"why":"Supplies the speech-to-text engine used to recognize spoken commands.","marker":"[7]"},{"why":"The prior plan recognition and trajectory prediction work that this framework builds on.","marker":"[11]"}],"fun_headline_variants":["Hierarchical robot framework cuts assembly time 15.9% in long-term tasks","Vision+speech robot framework boosts long-horizon collaboration success to 91.8%","Multimodal hierarchical framework improves human-robot teamwork by 15.9%","Speech plus vision helps robots predict human plans with 96% accuracy","Hierarchical plan prediction lifts robot success rate 14.22% over baseline"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole prediction chain assumes a complete, hand-written task graph that contains every plan the human might follow, along with fixed, known object locations; if a user follows a route the graph does not include, the robot has no reference sequence that matches and the prediction has no correct answer to find.","fun_headline_variants_meta":{"raw":{"variants":["Hierarchical robot framework cuts assembly time 15.9% in long-term tasks","Vision+speech robot framework boosts long-horizon collaboration success to 91.8%","Multimodal hierarchical framework improves human-robot teamwork by 15.9%","Speech plus vision helps robots predict human plans with 96% accuracy","Hierarchical plan prediction lifts robot success rate 14.22% over baseline"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000928,"raw_usage":{"total_tokens":3992,"prompt_tokens":980,"completion_tokens":3012,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":2907}},"tokens_in":596,"tokens_out":3012,"duration_ms":17039,"temperature":1.0,"reasoning_tokens":2907,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:58:52.179500+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same toy-car assembly with a participant instructed to follow an assembly order that is deliberately absent from the task graph, or place the tubes in a new location, and check whether plan predictions and task success stay at the reported 91.79% level; a significant drop would confirm that the manual-graph and fixed-object assumptions carry the stated result.","supporting_citations":[{"cited_title":"Homem de Mello and A.C","cited_arxiv_id":null,"evidence_quote":"Supplies the And-Or graph formalism used to structure the long-term task into alternative plan paths."},{"cited_title":"Matching incomplete time series with dynamic time warping: an algorithm and an application to post-stroke rehabilitation","cited_arxiv_id":null,"evidence_quote":"Provides the dynamic time warping algorithm used to match observed task progress to reference plan sequences."},{"cited_title":"Online model adaptation with feedforward compensation","cited_arxiv_id":null,"evidence_quote":"Supplies the online model adaptation method used to tune trajectory and action prediction to each user."},{"cited_title":"Are transformers effective for time series forecasting? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 11121–11128, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the DLinear time-series architecture used for trajectory and physical action prediction."},{"cited_title":"Deep speech 2: End-to-end speech recognition in english and mandarin","cited_arxiv_id":null,"evidence_quote":"Supplies the speech-to-text engine used to recognize spoken commands."},{"cited_title":"Towards efficient human-robot collaboration with robust plan recognition and trajectory prediction","cited_arxiv_id":null,"evidence_quote":"The prior plan recognition and trajectory prediction work that this framework builds on."}],"review_version":1}