{"id":"d81895ba-5951-4451-bba9-70d74c0117d6","arxiv_id":"2501.05936","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"MIAM, a multimodal dataset of 22 sessions of untrimmed industrial assembly and disassembly with RGB, depth, and IMU data, is released together with a fusion model that reports 86.79% accuracy on binary engagement prediction.","lead":"This paper introduces MIAM, a multimodal dataset of 22 sessions of untrimmed industrial assembly and disassembly videos with RGB, depth, and wearable IMU data, plus a fusion network for predicting whether an operator is engaged. It matters because realistic, multi-sensor industrial data could support human-robot collaboration and operator monitoring research.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Engagement-accuracy claim lacks a majority-class baseline and inter-annotator agreement; 86.79% on an 82.9% majority test set is not demonstrably an improvement.","rationale":"The reader's weakest assumption — that the engagement labels may not be valid ground truth — is indeed load-bearing, since noisy labels would undermine any accuracy figure. However, the most directly falsifiable hole is the evaluation protocol: with a 82.9% majority class and no baseline, error bars, or per-class metrics, the reported 86.79% cannot be interpreted as an improvement. Both issues point in the same direction and are fixable with additional analysis, so the reader's CONDITIONAL verdict remains appropriate. I do not see a fundamental flaw in the dataset's construction or the multimodal architecture; the missing evidence is statistical and annotation-quality related. Therefore the verdict should remain CONDITIONAL rather than moving to ACCEPT or REJECT.","tokens_in":6338,"tokens_out":3674,"duration_ms":39062,"concrete_test":"Release per-annotator engagement labels and compute Fleiss' kappa across the six annotators. Independently re-run the Table III evaluation adding an all-engaged majority baseline and reporting per-class F1 with bootstrap 95% confidence intervals, plus an RGB-only model. If kappa < 0.6 or the 86.79% accuracy is not significantly above 82.9% at p < 0.05, the engagement-improvement claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the fusion network 'improves the accuracy of recognizing engagement states' (Abstract) rests on Section III / Table III. The test set has 320 engaged vs 66 disengaged samples, so an all-engaged predictor already achieves 82.9%; the reported 86.79% is a 3.9-point gain with no majority-class baseline, no per-class precision/recall, no confidence intervals, and no repeated-run variance. Table III also omits an RGB-only row, so the marginal benefit of IMU/pose fusion cannot be isolated. The other load-bearing premise is label validity (Section II-C): six annotators, three per video, with 'Agreement was calculated by averaging scores' but no kappa or per-annotator agreement; resolving ambiguous cases in consensus can mask systematic disagreement. If the labels are noisy, the model may be fitting annotation artifacts rather than engagement. Action localization and object interaction are announced (Section II-C) but never evaluated, so those dataset contributions are unverified. These gaps are fixable, but as stated the empirical claim is not supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MIAM, a multimodal industrial activity monitoring dataset comprising 22 sessions (about 290 minutes) of untrimmed RGB, depth, and IMU data from assembly and disassembly tasks, together with action-level and engagement annotations. The authors also propose a multimodal fusion network that combines RGB features (I3D), IMU signals encoded as Gramian Angular Fields (ResNet-18), and MediaPipe skeleton embeddings through intra-modality attention, cross-modality co-attention, and gated fusion, and they report engagement prediction accuracy up to 86.79% on a held-out test set. The dataset is positioned as enabling action localization, object interaction, and engagement prediction in industrial human-robot collaboration.","tokens_in":6452,"tokens_out":4044,"duration_ms":42316,"significance":"If the dataset is made available and its annotations are validated, it could be a useful community resource: it combines multi-view RGB, depth, and wrist-mounted IMU data from realistic, untrimmed industrial workflows, and it includes fine-grained action taxonomies for assembly and disassembly. The proposed fusion network, although preliminary, provides a reasonable starting baseline for engagement prediction. The paper also makes the dataset and code publicly available, which is a strength for reproducibility. However, the current evaluation is too weak to support the paper's central empirical claims: the engagement experiment lacks a majority-class baseline, class-wise metrics, and error bars, and the other two advertised meta-tasks (action localization and object interaction) are not evaluated at all. The label-validity evidence is also incomplete because no inter-annotator agreement statistic is reported.","major_comments":[{"comment":"The central claim that the fusion network improves engagement recognition is not supported by the reported evaluation. The test set contains 320 engaged and 66 disengaged samples, so a trivial all-engaged classifier already achieves 82.9% accuracy. The reported 86.79% is only 3.9 percentage points above this baseline, and the paper provides no per-class precision/recall, F1 score, confusion matrix, confidence intervals, or repeated-run variance. Please add a majority-class baseline and report class-wise metrics and error bars; also define what a 'sample' is (clip length, stride, and how engagement labels are temporally assigned), since this is essential for interpreting the results.","section":"Section III / Table III"},{"comment":"Table III does not include an RGB-only row or any single-modality rows, so the marginal contribution of IMU and pose fusion cannot be isolated. The configurations listed all contain RGB plus at least one other modality, and the largest reported accuracy (RGB + LH + RH + Pose, 86.79%) might be driven primarily by the RGB stream. Please report RGB-only, IMU-only, pose-only, and all combinations in the ablation, along with the same evaluation protocol.","section":"Section III / Table III"},{"comment":"The validity of the engagement labels is not established. Six annotators labeled the videos, with three per video, but the paper only states that 'Agreement was calculated by averaging scores' and that ambiguous cases were resolved in consensus meetings. No inter-annotator agreement measure (e.g., Cohen's or Fleiss' kappa, or per-label agreement) is reported. Without such a measure, the 86.79% accuracy may reflect agreement with noisy or inconsistent annotations rather than with the true engagement state. Please report per-annotator agreement statistics, the label distribution, and the criteria used to resolve ambiguous cases.","section":"Section II-C"},{"comment":"Action localization and object interaction are presented as key meta-tasks supported by the dataset, and time-stamped annotations for them are claimed, but the paper provides no experiments, baseline evaluations, or even basic label statistics (e.g., number of action instances, object-interaction durations, per-class counts) for these tasks. As a result, the dataset contribution for these meta-tasks is unverified. Either add baseline evaluations and label statistics for these tasks, or explicitly limit the paper's claims to the dataset release and the engagement-prediction baseline.","section":"Section II-C / Section IV"},{"comment":"The experimental protocol is underspecified in ways that affect reproducibility. The paper states that camera C1 and the other sensors were synchronized with 'second-level timestamps', but for action localization with precise boundaries, second-level synchronization may be too coarse, and no synchronization accuracy or drift is reported. In addition, the engagement-prediction experiment omits training hyperparameters (optimizer, learning rate, batch size, epochs, number of runs), the temporal sampling scheme, and the exact definition of the train/test split at the sample level. Please provide these details.","section":"Section II-A / Section III"}],"minor_comments":[{"comment":"The sentence 'This process reflects real-world industrial workflows requiring precision, coordination, and the use of multiple tools.' appears twice verbatim in Section II-B; please remove the duplicate.","section":"Section II-B"},{"comment":"The I3D backbone is described as 'pre-trained on Kinetic [18]'; the dataset name should be 'Kinetics'.","section":"Section III"},{"comment":"The component list contains 'RPI head' and later 'rpi hat'; please standardize the terminology (hat/head) and use consistent capitalization for 'FRAS'.","section":"Section II-B"},{"comment":"Reference [11] is cited to support disengagement detection in human-robot dialogues, but the reference title is a survey on dialogue management; please verify that this citation is appropriate or replace it with a more specific reference.","section":"References"},{"comment":"The dataset involves human volunteers, but the paper does not state whether institutional ethics approval was obtained or whether informed consent was collected. For a human-subjects dataset, this information should be documented.","section":"Section II-B"}],"recommendation":"major_revision","confidential_remarks":"The paper is best framed as a dataset contribution, and the engagement-prediction experiment should be presented as an initial baseline rather than as a demonstration of a state-of-the-art method. The authors should also verify that the GitHub repository and dataset link are functional and that the license terms are clear before any final acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The dataset is the real contribution here. MIAM combines multi-view RGB-D with wearable IMU data in untrimmed industrial assembly and disassembly tasks, 290 minutes across 22 sessions, with step-level annotations. That combination is not in Meccano, HA4M, HRI30, or SenseCobot, and it gives the HRI/action-recognition community something they don't currently have: a realistic, multimodal industrial benchmark with engagement labels. The recording setup is sensible, and the annotation effort looks genuine, with six annotators and consensus for ambiguous cases.\n\nThe soft spots are all in the evaluation. The central claim that the fusion network improves engagement accuracy is not supported as stated. The test set is 82.9% engaged; an all-engaged predictor already gets 82.9%. Their best result, 86.79%, is 3.9 points above that, but there is no majority-class baseline reported, no confidence intervals, no repeated-run variance, and no per-class precision/recall. Table III also omits an RGB-only row, so you can't isolate the marginal contribution of IMU or pose. That is a fixable but serious omission.\n\nThe label validity also needs scrutiny. Six annotators, three per video, with \"agreement was calculated by averaging scores\" and no kappa or per-annotator agreement. Consensus meetings to resolve ambiguous cases can systematically hide disagreement. If the labels are noisy, the model may be fitting annotation artifacts rather than engagement. The second-level synchronization between camera and IMU clocks is probably fine for engagement states, which are longer than actions, but it is worth stating explicitly.\n\nAction localization and object interaction are announced as meta-tasks supported by the dataset but are never evaluated. Those dataset claims are unverified. Availability is also contradictory: the abstract gives a GitHub URL, the conclusion says the dataset \"will be made available\" on an institutional server, with no clear license or access steps in the paper.\n\nThese are all fixable. The dataset itself seems solid and well-documented, and the network is a reasonable first baseline, just not a validated one. This paper is for people working on industrial human-robot collaboration or action recognition in manufacturing. They will get value from the dataset, not from the network results. I would send it to peer review with a request for major revision: add a majority-class baseline, error bars, per-class metrics, inter-annotator agreement, evaluate or explicitly scope out the other meta-tasks, and clarify availability. The resource deserves referee time; the current empirical claim does not.","headline":"MIAM is a worthwhile dataset contribution, but the engagement-accuracy claim needs a majority-class baseline and inter-annotator agreement before it can be trusted.","tokens_in":7065,"tokens_out":2081,"would_cite":true,"duration_ms":21693,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces MIAM, a multimodal dataset of 290 minutes of untrimmed industrial assembly and disassembly video, and shows that fusing RGB, IMU, and skeleton data predicts operator engagement with 86.79% accuracy.","keywords":["multimodal dataset","industrial activity monitoring","engagement prediction","action localization","human-robot collaboration","assembly and disassembly","IMU sensor data","multimodal fusion"],"falsifier":"Re-annotate a subset of MIAM videos with a fresh group of annotators and compute Fleiss kappa against the published engagement labels; if agreement is below roughly 0.4, the ground truth is too noisy to support the reported 86.79% accuracy. A second check is to train the fusion model separately on each annotator's labels and see whether accuracy swings widely, which would show the averaged label hides disagreement.","tokens_in":6094,"feed_emoji":"🏭","tokens_out":8151,"duration_ms":72366,"temperature":0.7,"pith_summary":"The paper's aim is to give the human-robot collaboration field a dataset that reflects real, messy industrial work rather than staged clips. MIAM records eight volunteers performing a 91-step assembly and 62-step disassembly of a face-recognition attendance device across 22 sessions, totaling 290 minutes of continuous video with two camera views, depth, and hand-worn IMU signals. The same paper proposes a baseline network that fuses RGB, IMU, and skeleton pose, and reports that the fusion reaches 86.79% accuracy on binary engagement classification, with pose adding the largest single gain. If the dataset holds together, it provides a public benchmark for action localization, object interaction, and engagement prediction under realistic conditions.","feed_headline":"Fusing RGB, IMU, and pose hits 86.79% engagement accuracy","feed_subtitle":"MIAM dataset: 290 minutes of untrimmed assembly video with multi-view RGB, depth, and IMU for action and engagement benchmarks.","key_machinery":"The central object is the MIAM dataset itself: 22 sessions and 290 minutes of untrimmed video, captured by a 4K webcam, an Intel RealSense D455 RGB-D camera, and two 9-axis IMUs strapped to the operator's hands, all synchronized at second-level timestamps and logged to one PC. The carrying mechanism for the engagement result is the fusion network: I3D features from RGB clips, ResNet-18 features from Gramian Angular Field encodings of the IMU signals, and linearly embedded MediaPipe skeleton keypoints each pass through intra-modality attention, then cross-modality co-attention, and a gated fusion block yields the engaged/disengaged decision. The dataset's annotation pipeline, using the VIA tool with six annotators and consensus resolution for ambiguous cases, is what produces the ground truth for all three meta-tasks.","core_discovery":"The paper's central claim is that MIAM supplies what existing industrial datasets lack: synchronized multi-view RGB, depth, and hand IMU recordings of untrimmed assembly/disassembly workflows, annotated at action, object, and engagement levels. On the engagement meta-task, the claim is that a multimodal fusion network with intra-modality attention, cross-modality co-attention, and gated fusion improves accuracy as modalities are added, and specifically that full-body pose from MediaPipe produces the largest jump, with RGB plus pose reaching 85.75% and RGB plus both-hand IMU plus pose reaching 86.79%. The paper presents this as evidence that full-body context matters more than hand-worn motion sensors alone for recognizing engagement.","pith_inferences":["The paper reports only overall accuracy; because the Disengaged class has 131 training and 66 test samples against 1,441 and 320 Engaged, a model predicting only the majority class would already score about 82.9% on the test set, so per-class recall is needed to judge whether the 86.79% figure is genuine.","The stated second-level synchronization between camera and IMU clocks may be adequate for coarse action labels but too coarse for the exact moment engagement flips, so frame-level alignment is a natural extension.","Since the recording layout separates the operator workspace from a robot workspace and no robot interventions occur, a direct extension is to collect the same modalities during live human-robot collaboration, where disengagement labels are more consequential.","The GAF encoding of IMU acceleration and angular velocity could be evaluated on its own, without RGB or pose, to see how much wrist motion alone carries engagement information."],"forward_implications":["MIAM can serve as a benchmark for action localization and active object localization in continuous, untrimmed industrial video, where existing datasets are mostly trimmed or staged.","The ablation result implies that engagement-recognition systems gain the most from full-body pose, so future designs should weight skeleton streams over hand-only IMU.","Because the dataset includes natural disengagement events such as questions, distractions, and breaks, it enables work on detecting the moment of disengagement, not just classifying steady engagement.","The public release of synchronized RGB, depth, and IMU data under CC BY 4.0 makes the 70:30 video-level train/test split reproducible for later fusion models."],"supporting_citations":[{"why":"Provides the Meccano egocentric industrial dataset that MIAM positions itself against on multimodal coverage.","marker":"[6]"},{"why":"Supplies the HRI30 action recognition benchmark for industrial human-robot interaction that motivates action localization in MIAM.","marker":"[7]"},{"why":"Supplies the HA4M multi-modal assembly monitoring dataset, the closest prior assembly-task dataset for comparison.","marker":"[8]"},{"why":"Supplies the Enigma-51 dataset for fine-grained human behavior in industrial scenarios, a benchmark MIAM extends.","marker":"[9]"},{"why":"The VIA annotation tool used to label every video in MIAM.","marker":"[16]"},{"why":"I3D, the pretrained RGB feature extractor used in the fusion network.","marker":"[17]"},{"why":"Kinetics, the dataset used to pretrain the I3D RGB backbone.","marker":"[18]"},{"why":"ResNet-18, the pretrained encoder used on GAF-encoded IMU images.","marker":"[19]"},{"why":"ImageNet, the dataset used to pretrain the ResNet-18 IMU encoder.","marker":"[20]"},{"why":"MediaPipe, the skeleton extractor that supplies pose sequences to the network.","marker":"[21]"}],"fun_headline_variants":["Multimodal dataset tracks assembly tasks for engagement prediction","Fusing RGB, IMU, pose lifts engagement accuracy to 86.79%","MIAM dataset: 290 minutes of untrimmed assembly video for benchmarks","Pose data yields biggest gain in operator engagement recognition","Untrimmed multimodal video dataset enables action and engagement benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the six annotators' averaged binary engagement labels are valid ground truth; no inter-annotator agreement is reported, so if the labels are inconsistent, the 86.79% accuracy measures agreement with arbitrary annotation rather than real engagement.","fun_headline_variants_meta":{"raw":{"variants":["Multimodal dataset tracks assembly tasks for engagement prediction","Fusing RGB, IMU, pose lifts engagement accuracy to 86.79%","MIAM dataset: 290 minutes of untrimmed assembly video for benchmarks","Pose data yields biggest gain in operator engagement recognition","Untrimmed multimodal video dataset enables action and engagement benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00079,"raw_usage":{"total_tokens":3475,"prompt_tokens":928,"completion_tokens":2547,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":2458}},"tokens_in":544,"tokens_out":2547,"duration_ms":15723,"temperature":1.0,"reasoning_tokens":2458,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:05:55.557293+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a subset of MIAM videos with a fresh group of annotators and compute Fleiss kappa against the published engagement labels; if agreement is below roughly 0.4, the ground truth is too noisy to support the reported 86.79% accuracy. A second check is to train the fusion model separately on each annotator's labels and see whether accuracy swings widely, which would show the averaged label hides disagreement.","supporting_citations":[{"cited_title":"Meccano: A multimodal egocentric dataset for humans behavior understanding in the industrial- like domain,","cited_arxiv_id":null,"evidence_quote":"Provides the Meccano egocentric industrial dataset that MIAM positions itself against on multimodal coverage."},{"cited_title":"Hri30: An action recognition dataset for industrial human-robot interaction,","cited_arxiv_id":null,"evidence_quote":"Supplies the HRI30 action recognition benchmark for industrial human-robot interaction that motivates action localization in MIAM."},{"cited_title":"The ha4m dataset: Multi-modal monitoring of an assembly task for human action recognition in manufacturing,","cited_arxiv_id":null,"evidence_quote":"Supplies the HA4M multi-modal assembly monitoring dataset, the closest prior assembly-task dataset for comparison."},{"cited_title":"Enigma-51: Towards a fine-grained understanding of human behavior in industrial scenarios,","cited_arxiv_id":null,"evidence_quote":"Supplies the Enigma-51 dataset for fine-grained human behavior in industrial scenarios, a benchmark MIAM extends."},{"cited_title":"The via annotation software for images, audio and video,","cited_arxiv_id":null,"evidence_quote":"The VIA annotation tool used to label every video in MIAM."}],"review_version":1}