{"id":"f6bc4bb6-6a69-4e46-a262-69a1ab7b8b46","arxiv_id":"2411.11409","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new multimodal dataset aligns IKEA assembly videos with 3D part models, manuals, segmentation masks, and 6-DoF poses to benchmark 4D assembly understanding.","lead":"Researchers built the IKEA Video Manuals dataset, which links 3D furniture models, step-by-step manuals, and real-world assembly videos with frame-level annotations of part positions and motions. It gives AI systems a grounded benchmark for learning how to assemble furniture from watching videos.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"6-DoF pose ground truth rests on unvalidated camera estimation (§4.4): focal-length/depth coupling and missed camera changes can give near-perfect 2D reprojection with biased 3D poses; with no inter-annotator or calibrated-camera checks, the 4D-grounding claim and Table 4 numbers lack a noise floor.","rationale":"The reader and I converge on the same load-bearing point, so I did not manufacture a new objection. The reason this is the single most load-bearing concern rather than a peripheral one: the dataset's distinctive contribution is the spatio-temporal 4D alignment, and within it the 6-DoF pose annotations are the only modality that no prior internet-video assembly dataset provides (Table 1). The five application benchmarks all score against this GT. If the camera/pose estimation has a systematic bias, the central 'high-quality alignment' claim weakens, and the relative comparisons between baselines could be distorted, not merely shifted. Three concrete mechanisms (§4.4, App. C.6, Fig. A7) make the bias plausible rather than hypothetical: the f-tz coupling, the single-segment intrinsics assumption with manually detected camera changes, and the fact that every manual check ultimately compares against the same 2D imagery that is already consistent with a wrong 3D pose in the degenerate direction. At the same time, the paper has genuine independent support: the dataset and annotation interface are released with a DOI and GitHub repository, the annotation protocol is described at unusual length (App. C-E), the QC interface (App. D.2) checks masks, projections, and interpenetrations, and App. F provides seed-variation error bars for the two main benchmarks. None of these, however, quantifies GT accuracy. I also flag an internal overstatement: the NeurIPS checklist claims error bars are reported in the Appendix, but App. F covers only part-conditioned segmentation and 6D pose estimation; the assembly-plan (Table 2), VOS (Table 5), and shape-assembly (Fig. 7, Chamfer 0.33/0.55) results report single numbers without variance. This is secondary to the validation gap. My recommendation is CONDITIONAL rather than ACCEPT or REJECT: rejection would ignore a clearly valuable and carefully constructed resource, while unconditional acceptance leaves the flagship claim unmeasured where one quantitative experiment (inter-annotator or synthetic known-camera) could settle it.","tokens_in":22545,"tokens_out":13105,"duration_ms":139240,"concrete_test":"Validate the §4.4/C.6 pipeline against known camera parameters and independent annotations. (a) Create synthetic renders of the released 3D models with known intrinsics/extrinsics, furniture-like occlusions, and simulated camera changes; run the released estimation code and compare recovered vs. true intrinsics and per-frame poses. (b) Have a second annotator independently re-annotate ~200 frames from at least 10 videos and compute per-frame rotation/translation disagreement (ADD). Accept the annotations as benchmark-grade if median focal-length error is under 5% and inter-annotator ADD is below roughly 20% of the best Table 4 baseline ADD (MegaPose 1.36, i.e., <0.27); otherwise publish GT noise floors next to Tables 3-5 and re-check whether method rankings survive.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that IKEA Video Manuals provides high-quality 4D grounding, with per-frame 6-DoF part poses and per-segment camera intrinsics as the novel, distinguishing annotations (§3.1, Table 1). What must be true for this claim to hold is that those annotations are accurate enough to serve as benchmark ground truth. §4.4 and App. C.6 estimate intrinsics per video segment from manually labeled 2D-3D keypoints using PnP + RANSAC, then refine part poses manually. Three concrete failure modes threaten the assumption without being checked: (1) Focal-length/depth coupling: with intrinsics unknown and the camera roughly static, (f, tz) are weakly identifiable; a wrong focal length can be offset by a compensating translation along the optical axis, yielding near-perfect 2D reprojection — precisely the quantity the annotation interface displays — while the absolute 3D pose is biased. (2) A missed camera change (focal adjustment or camera switch) contaminates the intrinsics of the entire segment, shifting all poses in it. (3) Fig. A7 in App. C.9 explicitly shows that correct 2D projection coexists with incorrect 3D relative poses, and App. E.3's refinement re-checks against the same 2D video frame, so it cannot independently resolve the 3D ambiguity. No quantitative validation is reported: no inter-annotator agreement, no comparison with calibrated cameras or EXIF intrinsics, no reprojection-error statistics, no final-assembly closure residual. App. F reports error bars only for baseline models, not for the annotations. Since Tables 3-5 and Fig. 7 are all measured against this GT, the benchmark conclusions (e.g., MegaPose ADD 1.36 vs. SAM-6D 2.34) are interpretable only if GT noise is far below those errors; 'high-quality' is currently asserted rather than measured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces IKEA Video Manuals, a multimodal dataset that aligns 3D furniture models from IKEA-Manual with Internet assembly videos from IAW. For 98 videos and 36 furniture models it provides temporal step and substep alignments, per-frame part segmentation masks, per-frame 6-DoF part poses, and per-segment camera intrinsic estimates. The authors report five applications—assembly plan generation, part-conditioned segmentation, part-conditioned pose estimation, video object segmentation, and video-based shape assembly—with baseline evaluations showing that current methods perform poorly on this real-world data. The central claim is that this is the first dataset offering dense spatio-temporal (4D) grounding of assembly instructions in Internet videos.","tokens_in":22867,"tokens_out":2717,"duration_ms":29529,"significance":"If the annotation quality is adequate, this is a valuable benchmark contribution: it fills a real gap by combining real-world assembly videos with 3D models and manuals, and it includes a substantial annotation pipeline, detailed appendices, a datasheet, a public release under CC-BY-4.0, and a DOI. The five baseline tasks and the qualitative error analysis in Appendix H give the community concrete starting points. The paper's main weakness is that the novel 6-DoF pose and camera-intrinsic annotations are not validated quantitatively, which matters because the poses are the distinguishing asset of the dataset and are used as ground truth in Tables 4 and 5 and in Section 5.5.","major_comments":[{"comment":"The load-bearing assumption is that the per-segment camera intrinsics estimated from manual 2D-3D keypoints with PnP and RANSAC, followed by manual pose refinement, are accurate enough to serve as benchmark ground truth. This is not checked. The focal-length/depth coupling in uncalibrated monocular video means near-perfect 2D reprojection can coexist with biased 3D poses, and a missed camera change contaminates all poses in a segment. Fig. A7 explicitly shows that correct 2D projection does not imply correct 3D relative pose, yet the refinement interface in App. E.3 re-checks against the same 2D video frames. The manuscript reports no inter-annotator agreement, no comparison against calibrated cameras or EXIF intrinsics, no synthetic-reprojection validation, and no final-assembly closure residual. I would like to see a quantitative validation study, even on a subset, that establishes a noise floor for the pose annotations; without it, the 4D-grounding claim and the numbers in Table 4 rest on an unmeasured quantity.","section":"§4.4, App. C.6, Fig. A7"},{"comment":"The 'Error Bar' appendix reports standard deviations from running the evaluated models with three random seeds. This measures model stochasticity, not annotation uncertainty. For a dataset paper, the relevant error bar is the inter-annotator or inter-method variability of the ground-truth poses and masks. Since the pose annotations are the novel annotation type, I ask the authors to add annotation-level agreement statistics (e.g., re-annotating a subset by a second annotator and reporting pose/rotation errors and mask IoU).","section":"App. F"},{"comment":"The shape-assembly experiment in Setting 1 uses the annotated poses directly and reports a Chamfer Distance of 0.33. As written, this number conflates two things: the accuracy of the annotation pipeline and the quality of the assembly protocol. In particular, 'parts are not fully connected in the last frames of the substeps' is an annotation-consistency issue, not an algorithm failure. The discussion should separate annotation-induced error from method error, and ideally be tied to the validation requested above.","section":"§5.5"}],"minor_comments":[{"comment":"The sentence ending 'as illustrated in Fig. 4a.' is grammatically incomplete; it should be merged with the preceding sentence or rewritten.","section":"§4.1"},{"comment":"The 'IKEA in the Wild' row lists '# Object' as 420; please clarify whether this is the number of videos, the number of object instances, or the number of distinct object types, since the same row reports 14 object classes.","section":"Table 1 / Table A1"},{"comment":"The caption uses a full-width parenthesis in '（c) Parts in Frame Annotation'; please use consistent ASCII parentheses.","section":"Fig. 4 caption"},{"comment":"The symbol K is used for the number of parts in a sub-assembly in the assembly task, whereas K was earlier used for the number of parts in a plan-graph node in Section 5.1; please use distinct notation or define it again at first use.","section":"§5.5"},{"comment":"The field value 'firstPerson' in the example JSON should be 'first-person' for consistency with the Person View description.","section":"Fig. A2"},{"comment":"The Chamfer Distance values are reported without units or normalization. Please state whether the distance is in the model's native scale and how the ground-truth furniture is normalized across different furniture sizes.","section":"§5.5"}],"recommendation":"major_revision","confidential_remarks":"The paper is well within the scope of a datasets-and-benchmarks track and the dataset itself is a useful contribution. My recommendation hinges entirely on the missing quantitative validation of the pose ground truth. If the authors add a focused validation study—inter-annotator agreement, synthetic experiments, or camera-calibration cross-checks—I would be happy to support acceptance. I do not see a fundamental circularity or correctness error in the current manuscript; the issue is an unmeasured noise floor in the central annotation modality."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful dataset paper and the central resource is new. The gap it fills is real—IKEA-Manual has 3D models but no videos, Assembly101 and HA-ViD are lab capture, IAW has internet videos but no 3D poses. IKEA Video Manuals ties 3D models, manual steps, dense masks, and per-frame 6-DoF poses on 98 real internet assembly videos. That combination is new and worth having.\n\nThe paper does the dataset-construction legwork properly. The pipeline is detailed, the annotation interface is described with screenshots, the quality-control appendix lists concrete error modes (interpenetration, wrong part ID, SAM noise), and they ship code, data, a datasheet, and a license. The five applications and baseline evaluations are standard but fine for a datasets-and-benchmarks track; they show the data is challenging in ways that matter.\n\nThe soft spot is exactly the one the stress-test flags: the pose ground truth is only as good as the estimated camera intrinsics, and there is no independent accuracy check. Section 4.4 estimates intrinsics from manually labeled keypoints with PnP + RANSAC and then refines poses manually. The appendix openly shows a case where 2D projection looks right but the 3D relative pose is wrong (Fig. A7). That means the 'high-quality 4D grounding' is asserted, not measured. Missing: inter-annotator agreement, comparison against calibrated cameras or EXIF intrinsics, reprojection-error statistics, or any final-assembly closure residual. The stress-test's focal-length/depth coupling point is a legitimate concern, not a hypothetical: a wrong focal length with compensating translation can keep reprojection error near zero and bias absolute 3D pose. That matters for Table 4, because ADD scores of 1.36–3.33 are only interpretable if the GT noise floor is well below them. The paper should be required to add at least one form of external validation before publication.\n\nMinor: the shape-assembly experiment (Section 5.5) reports Chamfer distances without error bars, and the main-text segmentation/pose tables don't include the variance that App. F reports for some baselines. Both are easy fixes.\n\nNet: this deserves peer review, not desk rejection. The dataset is a real contribution and the paper is honestly written. I would accept with a request for independent pose-accuracy validation and a few error-bar additions. I'd also cite it in my own work on assembly benchmarks.\n\nWho it's for: anyone working on assembly understanding, 6-DoF pose estimation in the wild, VOS, or procedural video grounding. Bring it to reading group.","headline":"A useful, well-documented assembly dataset that deserves peer review; the pose ground truth lacks an independent accuracy check, and the paper should be revised to supply one.","tokens_in":23523,"tokens_out":2419,"would_cite":true,"duration_ms":57158,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"IKEA Video Manuals is the first dataset to ground assembly instructions in videos with dense spatio-temporal alignments among 3D part models, step-by-step manuals, and real-world internet videos.","keywords":["4D grounding","assembly dataset","6-DoF pose estimation","assembly instructions","video grounding","part segmentation","IKEA furniture","shape assembly"],"falsifier":"Take a random sample of annotated frames, project the annotated 3D part poses into the image using the provided camera intrinsics, and measure the silhouette overlap against the annotated segmentation masks; if the projected outlines systematically drift from the masks in frames with camera motion or across segment boundaries, the intrinsic estimation is biased. A stronger test is to independently calibrate a subset of videos using structure-from-motion on static background or known planar surfaces such as tabletops and compare the recovered intrinsics and poses; if the independent calibration disagrees with the dataset's estimates beyond a few degrees or centimeters, the claimed 6-DoF ground truth is not yet established.","tokens_in":22363,"feed_emoji":"🛠️","tokens_out":9419,"duration_ms":80469,"temperature":0.7,"pith_summary":"This paper introduces IKEA Video Manuals, a dataset built to connect every level of the furniture-assembly problem: the step-by-step instruction manual, the 3D geometry of the parts, and the messy real-world video of a person actually doing the assembly. Its central claim is that it is the first resource to provide 4D grounding of assembly instructions, meaning dense spatial and temporal alignment across these modalities, delivered as 34,441 annotated frames from 98 internet videos covering 36 IKEA furniture models. The paper argues that existing datasets each cover only a slice of the problem: some align manual steps without visual grounding, some provide 3D poses only in controlled labs, and internet video datasets lack correspondence with 3D models. If the dataset is sound, it gives the community a common benchmark on which task planning, visual grounding, and robot assembly can be studied together against real-world difficulties such as occlusions, changing viewpoints, and long assembly sequences, with five baseline tasks already defined.","feed_headline":"First dataset ties IKEA manuals, 3D parts, and video in 4D","feed_subtitle":"34,441 annotated frames from internet videos give machines a frame-by-frame map of how IKEA furniture goes together","key_machinery":"The central object is the 4D grounding itself: a dense spatio-temporal alignment in which 3D furniture parts, manual steps, and video frames are tied together, with each frame's sub-assemblies given a segmentation mask and a 6-DoF pose in the camera coordinate frame, plus per-segment camera intrinsics. The machinery that produces it is a semi-automated annotation pipeline: videos are split into substeps defined by the appearance of a new part or the formation of a sub-assembly; part identities are assigned by watching the whole video; masks are generated interactively with the Segment Anything Model and manually corrected; and poses come from hand-labeled 2D-3D keypoints solved with PnP and RANSAC to estimate intrinsics per video segment, then refined by annotators in a 3D interface that checks relative poses, coplanarity, and cross-frame consistency from multiple viewpoints. The load-bearing link is the mapping from each manual image to a contiguous span of video frames, so that the same assembly step has both a pictorial and a physical demonstration.","core_discovery":"On the paper's own terms, the discovery is the data itself: a benchmark in which each furniture item is represented by its 3D parts with 6-DoF poses, and each sampled video frame carries the identity, segmentation mask, and camera-frame 6-DoF pose of every sub-assembly in view, together with camera intrinsics estimated per video segment and a mapping from each manual step to a span of video substeps. The authors claim this is the first dataset to supply 6-DoF pose annotations for furniture assembly from internet videos, combining the 3D models and manuals of the IKEA-Manual dataset with in-the-wild part trajectories that prior datasets only approximated. The annotation process assembles these correspondences through temporal segmentation into substeps, part identity tracking, Segment Anything Model assisted mask annotation, keypoint-based camera estimation with PnP and RANSAC, and manual 3D refinement verified from multiple viewpoints. On this resource the paper defines five applications, assembly plan generation, part-conditioned segmentation, part-conditioned pose estimation, video object segmentation, and furniture assembly from instruction videos, and reports baselines showing that all of them remain difficult.","pith_inferences":["The camera intrinsics estimated per video segment are the hidden load-bearing assumption: the paper states that correct camera parameters are a prerequisite for pose accuracy, yet reports no independent calibration check or inter-annotator agreement study, so a systematic bias in those estimates would transfer directly into every 6-DoF pose number in the benchmark.","Because the annotation pipeline is built on Segment Anything Model prompts plus keypoint PnP, the marginal cost of extending the dataset to new furniture categories is plausibly much lower than for fully manual datasets; the same pipeline could be reapplied to other branded assembly products.","All pose baselines run from monocular RGB with estimated depth, even though the dataset supplies true 3D geometry and camera parameters; a natural next experiment, not run in the paper, is to measure how much of the reported gap closes when methods are allowed to use the provided intrinsics and multi-frame consistency.","The manual refinement step that corrects 2D-projection errors by checking part poses from top and side views suggests that pure automatic pose recovery from keypoints is insufficient for ground-truth quality, which implies the dataset's accuracy depends on human effort that future scaling of the resource will need to control."],"forward_implications":["Because the dataset anchors manual steps, 3D poses, and video pixels to the same events, a system that masters its five benchmark tasks could watch an assembly video and produce an executable assembly plan grounded in real object poses rather than in 2D annotations alone.","The reported baselines show that zero-shot part segmentation and pose estimation methods such as CNOS, SAM-6D, and MegaPose degrade substantially on internet video compared with lab-based benchmarks, quantifying a measurable gap between current methods and in-the-wild assembly understanding.","The video-based assembly task demonstrates that with ground-truth keyframe and pose annotations a modular pipeline assembles furniture to a Chamfer distance of 0.33, while replacing keyframe detection with GPT-4o degrades this to 0.55, indicating that automatic step detection is currently the weakest link in video-guided assembly.","The dataset's comparison with existing resources such as Assembly101, IKEA ASM, IKEA-Manual, and IKEA in the Wild positions it as the only one combining internet-sourced video, 3D object models, and dense 6-DoF annotations, enabling evaluation of generalization across environments and assembly sequences."],"supporting_citations":[{"why":"Supplies the 36 segmented 3D furniture models and manual-step annotations that the new dataset aligns with videos.","marker":"[2]"},{"why":"Supplies the 98 internet assembly videos and the coarse step segmentation that the paper refines into fine-grained substeps.","marker":"[32]"},{"why":"The Segment Anything Model powers the interactive mask annotation: prompt points generate initial part masks that annotators then correct.","marker":"[33]"},{"why":"The PnP algorithm used to estimate object poses and candidate camera intrinsics from the annotated 2D-3D keypoint correspondences.","marker":"[34]"},{"why":"RANSAC filters outlier keypoints during camera intrinsic estimation, making per-segment intrinsic selection robust.","marker":"[35]"},{"why":"One of the lab-based RGB-D assembly datasets with calibrated cameras that the paper contrasts against when claiming to be the first with 6-DoF poses on internet videos.","marker":"[3]"},{"why":"Assembly101 is the largest existing multi-view assembly dataset; its calibrated-lab setup is the main point of comparison in the dataset-positioning table.","marker":"[11]"},{"why":"MegaPose is the strongest pose-estimation baseline evaluated, setting the reference point for the benchmark's difficulty.","marker":"[39]"}],"fun_headline_variants":["First 4D dataset grounds IKEA assembly instructions in videos","IKEA Video Manuals: 4D grounding for assembly from real clips","4D mapping of IKEA assembly from 3D parts to video frames","New benchmark ties IKEA manuals, 3D parts, and video in 4D","Grounding assembly steps in 4D: IKEA Video Manuals dataset"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The dataset's pose annotations are treated as ground truth, but the camera parameters they depend on are estimated from hand-labeled keypoints and manual refinement rather than measured or independently validated, so any systematic error in the cameras would silently bias every downstream pose number.","fun_headline_variants_meta":{"raw":{"variants":["First 4D dataset grounds IKEA assembly instructions in videos","IKEA Video Manuals: 4D grounding for assembly from real clips","4D mapping of IKEA assembly from 3D parts to video frames","New benchmark ties IKEA manuals, 3D parts, and video in 4D","Grounding assembly steps in 4D: IKEA Video Manuals dataset"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000327,"raw_usage":{"total_tokens":1846,"prompt_tokens":980,"completion_tokens":866,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":770}},"tokens_in":596,"tokens_out":866,"duration_ms":7921,"temperature":1.0,"reasoning_tokens":770,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:31:43.443054+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of annotated frames, project the annotated 3D part poses into the image using the provided camera intrinsics, and measure the silhouette overlap against the annotated segmentation masks; if the projected outlines systematically drift from the masks in frames with camera motion or across segment boundaries, the intrinsic estimation is biased. A stronger test is to independently calibrate a subset of videos using structure-from-motion on static background or known planar surfaces such as tabletops and compare the recovered intrinsics and poses; if the independent calibration disagrees with the dataset's estimates beyond a few degrees or centimeters, the claimed 6-DoF ground truth is not yet established.","supporting_citations":[{"cited_title":"Ikea-manual: Seeing shape assembly step by step","cited_arxiv_id":null,"evidence_quote":"Supplies the 36 segmented 3D furniture models and manual-step annotations that the new dataset aligns with videos."},{"cited_title":"Aligning step-by-step instructional diagrams to video demonstrations","cited_arxiv_id":null,"evidence_quote":"Supplies the 98 internet assembly videos and the coarse step segmentation that the paper refines into fine-grained substeps."},{"cited_title":"Ep n p: An accurate o (n) solution to the p n p problem","cited_arxiv_id":null,"evidence_quote":"The PnP algorithm used to estimate object poses and candidate camera intrinsics from the annotated 2D-3D keypoint correspondences."},{"cited_title":"Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography","cited_arxiv_id":null,"evidence_quote":"RANSAC filters outlier keypoints during camera intrinsic estimation, making per-segment intrinsic selection robust."},{"cited_title":"The ikea asm dataset: Understanding people assembling furniture through actions, objects and pose","cited_arxiv_id":null,"evidence_quote":"One of the lab-based RGB-D assembly datasets with calibrated cameras that the paper contrasts against when claiming to be the first with 6-DoF poses on internet videos."},{"cited_title":"Assembly101: A large-scale multi-view video dataset for understanding procedural activities","cited_arxiv_id":null,"evidence_quote":"Assembly101 is the largest existing multi-view assembly dataset; its calibrated-lab setup is the main point of comparison in the dataset-positioning table."}],"review_version":1}