{"id":"ee605115-d709-4d61-828d-3a91f6ea8b6c","arxiv_id":"2504.21033","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A system that uses Mask R-CNN to isolate objects in an AR headset view and Shap-E to generate their 3D models, with a reported System Usability Scale score of 69.64.","lead":"This paper builds an augmented reality tool that turns a photo of an object into a 3D model you can see through the headset. It combines an object-detection step with OpenAI's Shap-E model, and a 35-person test gave it a middling usability score.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never validates that the final Shap-E mesh matches the intended object, so the central 'seamless generation' claim is unsupported; the real-time claim is also contradicted by the reported 43.2 s conversion.","rationale":"The reader correctly identified Mask R-CNN detection as a weak link, but the more load-bearing gap is downstream: even a perfect detection and crop does not guarantee a usable 3D model. Shap-E is known to produce plausible but often incorrect geometry from single images, and the paper's own Figure 2 demonstrates exactly this failure for a mug. The user-verification menu in Section 4.5 only validates the detected object, not the generated mesh, so the central 'easily generate... seamlessly integrated' claim is unsupported by any quality measurement. Separately, the paper's performance table contradicts the 'real-time' in the title and abstract: the sum of reported timings is roughly 68 seconds, and no end-to-end latency or interaction latency is measured. Both issues were already reflected in the reader's conditional verdict, so I do not recommend changing the verdict. The condition should explicitly require objective model-fidelity evaluation and either a redefinition or removal of the real-time claim.","tokens_in":10985,"tokens_out":8285,"duration_ms":83823,"concrete_test":"Run the full pipeline on 20 objects for which ground-truth 3D meshes are available (e.g., ShapeNet categories), photographing each in an AR-like scene with background clutter. Compute Chamfer distance, volumetric IoU, and normal consistency between the simplified Shap-E output and the ground-truth mesh, and have three independent raters name the object from the generated mesh. Report the median and maximum errors and the recognition rate; if recognition falls well below the raters' agreement on the original photographs, the 'seamless generation' claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central usability claim depends on the assumption that, after Mask R-CNN isolation and cropping, the image sent to Shap-E yields a usable 3D model of the intended object. The paper provides no objective evidence for this: no reconstruction accuracy metric, no user rating of model fidelity, no success/failure rate. Section 4.5's user verification occurs before generation and only confirms the detected object's label; it cannot catch a wrong Shap-E output. The paper's own Figure 2d shows Shap-E turning a mug image into a laptop, and the workflow has no post-generation verification step. Even if the reader's detection concern is fully resolved by segmentation, generation fidelity remains unmeasured. A secondary but separately decisive issue is that Table 1 reports 5.2 s detection + 43.2 s conversion + 9.1 s simplification + 10.3 s load/render, roughly 68 s total, directly contradicting the abstract's 'real time' claim. Either gap alone leaves the central claim without support.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes an AR system that lets a HoloLens 2 user select a region or request all objects, applies Mask R-CNN to detect and crop objects, lets the user confirm detected labels, sends the cropped image to Shap-E for 3D mesh generation, simplifies the mesh, and renders it in AR. The evaluation consists of a timing table (Table 1) and a SUS usability study of 35 participants with an ANOVA split by AR/VR familiarity. The authors claim real-time generation, seamless image-to-3D conversion, and easy 3D model creation in AR as their primary contributions.","tokens_in":11167,"tokens_out":4382,"duration_ms":42413,"significance":"If the claims were supported, the value would be a practical integration of well-known components into an AR workflow, with a credible usability signal: the SUS score of 69.64 and the significant familiarity effect (F=18.212, p<0.001) are analyzed and reported clearly. The paper is also honest in showing Shap-E failure modes in Figure 2, which is a strength. However, the lack of any objective fidelity metric and the direct contradiction between the 'real-time' claim and the reported timings mean the central claims are not yet established. As it stands, the contribution is an integration demo with a usability study, not a validated real-time generation system; the use of off-the-shelf components aids reproducibility in principle.","major_comments":[{"comment":"The abstract and Section 4 claim that users can generate and interact with 3D models 'in real time' and that the primary contribution is a 'seamlessly integrated' process, but Table 1 reports Image Processing for Object Detection Time of 5.2 s, Image-to-3D Conversion Time of 43.2 s, Model Simplification Time of 9.1 s, and Load and Render Time of 10.3 s, totaling approximately 67.8 s per object. This directly contradicts the real-time claim, and the table provides single measurements without variance, test conditions, or sample size. The authors should either revise the real-time/interactive wording throughout or provide evidence of a genuinely real-time configuration.","section":"Abstract; Section 4; Table 1"},{"comment":"The central claim that the system converts 2D images into usable 3D models is not supported by any objective evaluation of generation fidelity. Section 4.5's user verification occurs before Shap-E generation and only confirms the detected object's class label; it cannot catch a wrong or degenerate Shap-E output. Figure 2d itself shows Shap-E producing a laptop from a mug image, and Section 4.6 only displays the generated model without any accuracy check. The paper needs metrics such as reconstruction accuracy, human ratings of model-object correspondence, or a success/failure rate on a defined test set, plus a post-generation verification step, to support the 'seamless transformation' claim.","section":"Section 4.5-4.6; Figure 2"},{"comment":"The pipeline's first AI stage, Mask R-CNN with COCO pretrained weights and a 0.5 confidence threshold, is never evaluated on AR headset imagery. Section 4.3 describes the configuration but reports no detection accuracy, precision/recall, or failure rate for the actual capture conditions, so it is unknown how often the cropped input to Shap-E contains the intended object. The performance section measures only processing time, not detection correctness. If detection fails or mislabels, the user-verification menu in Section 4.5 either shows no object or the wrong object, and no usable 3D model can be generated; the authors should quantify this failure mode.","section":"Section 4.3; Section 5.2"},{"comment":"The SUS study is analyzed correctly, but it measures only perceived usability, not whether the generated 3D models match user intent. There are no task-success rates, model-quality ratings, or completion times, so the usability score cannot support the claim that users can 'easily generate... 3D models' of their intended objects; a system can be usable yet produce unusable output. The authors should add a task-based evaluation that records generation outcomes alongside the SUS questionnaires.","section":"Section 5.3-5.4"}],"minor_comments":[{"comment":"The first paragraph reads 'notable advancements natural language processing' and is missing the preposition 'in'.","section":"Section 1"},{"comment":"The header 'Measured V alue' contains a spacing typo; it should read 'Measured Value'.","section":"Table 1"},{"comment":"The description of SUS scoring says 'subtracting 1 from score' and 'subtracting score from 5'; it should say 'subtracting 1 from the score' and 'subtracting the score from 5' for grammatical clarity.","section":"Section 5.4"},{"comment":"The text references 'The boxplot in Figure 13' but no Figure 13 is included in the manuscript; all referenced figures should be present and numbered consistently.","section":"Figure 13"},{"comment":"Reference 30 has a typo: 'Porceedinsg' should be 'Proceedings'.","section":"References"},{"comment":"The recommended hardware specifies a GPU with 16 GB of memory, while Table 1 reports 6.8 GB GPU memory consumption; please clarify whether the 16 GB recommendation is for the full system or a different hardware configuration.","section":"Section 6.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's novelty is incremental, being an integration of off-the-shelf components; this is acceptable for a systems/usability venue only if the claims are matched to the evidence. The main risk is the unsupported 'real-time' and fidelity claims, which I would not treat as a mere wording issue: they affect the title, abstract, and the stated primary contribution. A revision that adds a fidelity evaluation, reports detection success rates, and recalibrates the real-time language could make the contribution solid, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is an incremental integration of known components, but it is a real, working system with a properly conducted usability study. The workflow—lasso selection, Mask R-CNN cropping, user verification, Shap-E generation—is clearly described and directly addresses the failure modes shown in their own Figure 2. That honest presentation of Shap-E's limitations is a point in the paper's favor.\n\nThe soft spots are real and not minor. The abstract says \"real time,\" but Table 1 reports 43.2s for image-to-3D conversion and roughly 68s end-to-end. That is not real-time by any reasonable AR standard. Either the claim needs to be dropped or the system needs to be run on faster hardware, but the paper as written contradicts itself. More importantly, the central claim—that users can seamlessly generate usable 3D models—is never validated. The user verification step only confirms the detected object's label before generation; it cannot catch a wrong Shap-E output. No reconstruction accuracy metric, no user rating of whether the generated model actually resembles the source object, no success/failure rate. The paper's own Figure 2d shows Shap-E turning a mug into a laptop, and the pipeline has no post-generation check. That gap leaves the headline capability unsupported.\n\nThe usability study is the strongest part. The SUS scoring is correct, the ANOVA is properly computed, and the split between frequent and rare AR users is a reasonable way to analyze experience effects. But the result—that people who already like AR rate it higher—is predictable and doesn't tell us much about the system's quality.\n\nIf this paper were reframed as a systems note—dropping \"real-time,\" adding an objective or subjective fidelity evaluation, releasing code or at least more detailed performance data—it would be a useful contribution for HCI and AR researchers. As it stands, the hype in the abstract mismatches the evidence. I would send it to peer review only if the venue tolerates integration papers with modest scientific claims and heavy revision; otherwise it's a desk reject. The ideas are clear and honestly motivated, but the evaluation is incomplete.","headline":"A plausible AR image-to-3D integration with a solid usability study, but the real-time claim is contradicted by its own 43.2s conversion and the final mesh quality is never validated.","tokens_in":11733,"tokens_out":1707,"would_cite":false,"duration_ms":19389,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that a lasso-and-verify AR workflow can turn 2D snapshots into 3D models for non-experts.","keywords":["generative AI","augmented reality","image-to-3D generation","Shap-E","Mask R-CNN","object detection","System Usability Scale","3D model generation"],"falsifier":"Run the full capture-to-model workflow on a set of fifty real headset scenes covering cluttered desks, monitors displaying product photos, and objects under varied lighting, and record how often the verification menu contains the user's intended object and how often the Shap-E output is recognized as resembling it; if the menu is empty or wrong in most trials, or the meshes rarely resemble the targets, the central usability claim fails.","tokens_in":1444,"feed_emoji":"🥽","tokens_out":6373,"duration_ms":154372,"temperature":0.7,"pith_summary":"Traditional 3D modeling requires skill and time. The paper tries to establish that a headset-based workflow built from existing components can let non-specialists turn 2D snapshots into 3D models. The user draws a lasso around a region or asks to capture everything; Mask R-CNN isolates and labels objects; a verification menu rejects wrong detections; and Shap-E converts the accepted crop into a 3D mesh shown back in the AR scene. This matters because raw Shap-E fails on cluttered scenes and on monitors, producing a cube or a laptop instead of the intended mug. The paper supports the claim with a 35-person study giving an overall System Usability Scale score of 69.64 and a statistically higher 80.71 for frequent AR/VR users ($F = 18.21$, $p < 0.001$).","feed_headline":"AR pipeline turns 2D photos into 3D models for non-experts","feed_subtitle":"No 3D modeling skill needed: capture, verify, and place an object in augmented reality.","key_machinery":"The carrying mechanism is the insertion of a detection-and-verification stage between the raw camera image and the generative model. Mask R-CNN, an instance-segmentation model pre-trained on the COCO dataset with a confidence threshold of 0.5, separates objects from complex backgrounds and labels them; the user then confirms or rejects these detections in a menu before any 3D generation runs. Shap-E, a generative model that outputs a 3D mesh from an image, receives only the accepted, tightly cropped object. This arrangement is what converts Shap-E's documented failures on cluttered or screen-displayed scenes into a usable, interactive pipeline, and the reported latencies are the evidence behind the real-time claim.","core_discovery":"The central claim, stated as the paper's primary contribution, is a process for converting 2D images into 3D models that can be integrated into AR environments, rather than a new AI model. The paper documents two failure modes of the raw generator: a multi-object desk scene collapses into a single cube, and a mug photographed from an online store becomes a laptop. The proposed process addresses these failures by interposing object isolation and human verification: zone selection, cropping, Mask R-CNN instance segmentation, edge-based object cropping, a user menu for selecting detected objects, and finally Shap-E generation and rendering in the headset view. Measured latencies are reported for each stage: image processing 5.2 seconds, image-to-3D conversion 43.2 seconds, model simplification 9.1 seconds, load and render 10.3 seconds, with 61 percent average GPU utilization. The user study found a statistically significant difference in usability scores between infrequent and frequent AR/VR users, with means of 64.38 and 80.71 and $F = 18.21$, $p < 0.001$.","pith_inferences":["Because the detector is COCO-trained, the object vocabulary is bounded to common classes; fine-tuning on store-specific or headset-specific objects would be an evident next step that the paper does not report.","The conversion stage is a pluggable bottleneck: if Shap-E is replaced with a newer image-to-3D generator, the same crop-and-verify front end should transfer, suggesting that object isolation, not the generator, is the main contribution.","The user-verification step could become a data-collection loop in which accepted crops form labeled pairs for retraining the detector under headset lighting and cluttered backgrounds.","A natural stress test is sequential generation: with conversion taking 43.2 seconds per object, converting several objects in one scene will reveal whether the interaction remains acceptable."],"forward_implications":["A user without 3D modeling skill can capture a real-world object or an on-screen product photo and receive a 3D mesh placed back into the AR scene.","The process transfers to VR and other immersive environments, because the image-to-3D conversion stage is independent of the AR front end.","The applications named by the paper, including gaming, education, interior design, and e-commerce, become practical without dedicated modeling staff.","The verification menu prevents the system from spending GPU time converting objects the user did not intend.","Novice users are the group most likely to need interface simplification, since frequent AR/VR users rated usability significantly higher."],"supporting_citations":[{"why":"This is the paper's in-text citation for the explicit 3D representation underlying Shap-E; the generator builds meshes from the accepted cropped image.","marker":"[1]"},{"why":"It supplies the Mask R-CNN instance-segmentation method used to detect and separately crop objects within the selected AR view.","marker":"[14]"},{"why":"It supplies the lasso-style zone-selection interaction technique used to delimit which objects in the headset view are captured.","marker":"[29]"}],"fun_headline_variants":["AI-powered AR turns 2D snapshots into 3D models on the fly","From photo to 3D: AR workflows with generative AI for all","Snap it, select it, place it: AI generates 3D models in AR","Generative AI bridges 2D and 3D for AR creation without expertise","Real-time 3D model generation from images in augmented reality"],"cache_read_input_tokens":13952,"weakest_assumption_plain":"The entire workflow assumes the pre-trained object detector will find and segment the object the user intends; the paper reports no detection accuracy or failure rate, so if the detector misses or mislabels the target, the verification menu offers no usable object and no 3D model gets generated.","fun_headline_variants_meta":{"raw":{"variants":["AI-powered AR turns 2D snapshots into 3D models on the fly","From photo to 3D: AR workflows with generative AI for all","Snap it, select it, place it: AI generates 3D models in AR","Generative AI bridges 2D and 3D for AR creation without expertise","Real-time 3D model generation from images in augmented reality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000538,"raw_usage":{"total_tokens":2589,"prompt_tokens":960,"completion_tokens":1629,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":1526}},"tokens_in":576,"tokens_out":1629,"duration_ms":11705,"temperature":1.0,"reasoning_tokens":1526,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:54:56.676181+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full capture-to-model workflow on a set of fifty real headset scenes covering cluttered desks, monitors displaying product photos, and objects under varied lighting, and record how often the verification menu contains the user's intended object and how often the Shap-E output is recognized as resembling it; if the menu is empty or wrong in most trials, or the meshes rarely resemble the targets, the central usability claim fails.","supporting_citations":[{"cited_title":"Learn- ing representations and generative models for 3D point clouds","cited_arxiv_id":null,"evidence_quote":"This is the paper's in-text citation for the explicit 3D representation underlying Shap-E; the generator builds meshes from the accepted cropped image."},{"cited_title":"CAST: Effective and efficient user interaction for context-aware selection in 3D particle clouds","cited_arxiv_id":null,"evidence_quote":"It supplies the lasso-style zone-selection interaction technique used to delimit which objects in the headset view are captured."}],"review_version":1}