{"id":"2a0d1fbe-23d0-4fd6-a8ae-e4c13304f906","arxiv_id":"2412.07487","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"StereoHO combines two RGB views of a hand holding an object into a joint 3D reconstruction via learned shape codebooks, enabling robots to grasp diverse household objects, including transparent ones.","lead":"This paper presents StereoHO, a method that reconstructs the 3D shape of a hand and a held object from two ordinary RGB camera views, using learned shape priors and a probabilistic combination of the two views. The reconstructed shape is used to guide a robot arm to safely receive objects, including transparent glasses that depth cameras miss.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains over IHOI and SVHO may be confounded by StereoHO's use of segmentation masks as additional input; the paper's own ablation shows masks improve object Chamfer distance.","rationale":"The reader identified the conditional-independence assumption of the stereo fusion rule as the weakest assumption, but that assumption is not actually load-bearing for the paper's central comparative claim. The method could work well in practice even if the independence assumption is violated, and the paper does not present the independence claim as a theoretical guarantee; it is a design heuristic. The more load-bearing issue is the fairness of the quantitative comparison: StereoHO uses segmentation masks as input, and the paper's own ablation shows that masks materially improve object reconstruction. Since IHOI and SVHO are standard RGB-only methods in the comparison, the reported Chamfer distance reductions may be attributable to the privileged mask input rather than the proposed stereo probabilistic fusion. This directly threatens the abstract's claim that the method reduces object Chamfer distance compared to existing RGB-based methods. The reader's concerns about missing error bars and the handover comparison with CB are real but secondary; the mask confound is the clearest threat to the headline result. Therefore the verdict remains CONDITIONAL, but for a different primary reason: the evaluation must control for input modality before the superiority claim can be accepted.","tokens_in":12961,"tokens_out":8630,"duration_ms":84396,"concrete_test":"Run the DexYCB reconstruction evaluation under two matched conditions: (1) evaluate StereoHO without segmentation masks (using the already-reported 'Ours (no segmentation mask)' variant) against IHOI and SVHO, also without masks; and (2) retrain or adapt IHOI and SVHO to accept the same segmentation masks as extra input (or append the mask channels to their RGB input) and re-run the comparison. If the object Chamfer distance margins over the baselines shrink to below a meaningful threshold (e.g., 5 cm²) in either condition, the claim of improvement due to the stereo-fusion and codebook approach would need to be downgraded; if the margins remain large in the no-mask comparison, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central quantitative claim is that StereoHO reduces object Chamfer distance compared to RGB baselines (IHOI single-view, SVHO stereo). However, StereoHO's input includes hand and object segmentation masks (predicted or possibly ground-truth) in addition to RGB crops, while IHOI and SVHO are evaluated with RGB only. The paper's own ablation (Section IV-D, Fig. 7, 'Ours (no segmentation mask)') demonstrates that adding masks improves object reconstruction, meaning a substantial part of the reported gap over baselines could stem from this extra input rather than from the probabilistic stereo fusion or the learned codebook. The text does not state whether masks are ground-truth or predicted during DexYCB evaluation, nor whether the baselines were provided any mask information. Because the abstract's headline claim is an apples-to-apples comparison with existing RGB methods, the absence of a controlled input modality makes the improvement not yet attributable to the proposed technical components. The independence assumption raised in the reader's verdict is secondary: even if per-view distributions were conditionally independent, the mask input confound would still undermine the comparison.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes StereoHO, a stereo RGB method for joint hand-object reconstruction that learns discrete codebooks of shape embeddings on synthetic data, predicts per-view probability distributions over these codebooks, and fuses the two views by element-wise multiplication of the distributions before decoding to a truncated signed distance field and pointcloud. The reconstructed hand-object shape is filtered by projecting it against predicted segmentation masks, and the output is used to plan 6-DoF grasps for a UR5 robot in human-to-robot handovers. The authors evaluate reconstruction on DexYCB against IHOI (single-view) and SVHO (stereo), reporting lower object Chamfer distances, and conduct real handover experiments with containers and general household objects, including transparent ones. The central claims are that the probabilistic combination of single-view predictions improves object reconstruction over existing RGB-based single-view and stereo methods, and that the reconstructed shapes enable successful robot handovers.","tokens_in":13222,"tokens_out":4256,"duration_ms":42761,"significance":"If the central claims hold, the paper makes a useful contribution: it is among the first to combine learned codebook priors with a probabilistic multi-view fusion for hand-object reconstruction from RGB only, and it demonstrates a complete handover pipeline with real robot experiments and transparent objects. The use of synthetic-only training with evaluation on real data (DexYCB and physical handovers) is a strength, and the paper ships a reproducible link to code and videos. The reported object Chamfer improvements over IHOI and SVHO are potentially large, and the real-robot success rates (75-91%) for household objects are encouraging. However, the comparison is confounded by the use of segmentation masks as additional input, and the paper provides no statistical uncertainty quantification; these issues currently limit the strength of the claims.","major_comments":[{"comment":"The central quantitative comparison is confounded by an input modality difference. Section III-A states that the model takes the segmentation masks of the object (MO) and hand (MH) as input, and the ablation 'Ours (no segmentation mask)' in Fig. 7 shows that masks improve object reconstruction. The paper does not state whether masks are ground-truth or predicted during DexYCB evaluation, nor whether the baselines IHOI and SVHO received any mask information. Because the abstract claims a reduction in object Chamfer distance compared to 'existing RGB based hand-object reconstruction methods,' the reported gains cannot be attributed to the proposed codebook and probabilistic fusion until the comparison is controlled for the additional mask input. The authors should either report the no-mask variant as the main result, provide baselines augmented with masks, or explicitly justify why mask input is not an unfair advantage in the comparison.","section":"Section III-A and Section IV-D (Fig. 7)"},{"comment":"The stereo fusion rule assumes that the per-view probability distributions over the codebook are conditionally independent given the true hand-object shape. This assumption is not validated, and the paper provides no alternative fusion rules (e.g., sum, max, or a learned combination) or any calibration analysis. Without such an ablation, the claim that the method combines single-view reconstructions 'probabilistically' to form a coherent stereo reconstruction is not fully supported: the stereo improvement over SVHO could be driven by the additional view, the mask input, or the codebook representation rather than by the product rule. A controlled comparison of fusion rules, or evidence that PL and PR are calibrated for the reported improvement, is needed.","section":"Section III-A, equation P = PLPR"},{"comment":"No error bars, confidence intervals, or significance tests are reported for any reconstruction or handover metric. The reported differences, such as the 9.71 cm^2 object Chamfer improvement over IHOI in the single-view seen category, may be within run-to-run or subject-to-subject variation; the paper does not specify the number of DexYCB frames used or the variance across participants. The handover success rates (e.g., 75% and 91% on household objects) are aggregates over configurations with no per-participant breakdown or interval estimates. The authors should report standard deviations across multiple training runs or test splits, and ideally a paired statistical test, to support the quantitative claims.","section":"Section IV-D and Section IV-E (Table I, Fig. 7)"},{"comment":"The handover comparison with the CORSMAL baseline (CB) is not apples-to-apples: CB assumes the object starts on the table and remains upright, while StereoHO assumes the object is already held by the human. The text concludes the methods are 'on par' for containers, but the differing assumptions and the lack of statistical testing make this conclusion difficult to interpret. The authors should either align the test conditions or explicitly acknowledge that the comparison is qualitative rather than a controlled benchmark.","section":"Section IV-E, comparison with CB"}],"minor_comments":[{"comment":"Typo: 'Signed Distance Dields' should be 'Signed Distance Fields.'","section":"Section II"},{"comment":"The notation 'TW B' is ambiguous; please clarify the subscript/superscript convention for the wrist-to-robot transform, or use a consistent bold/italic format.","section":"Section III-D"},{"comment":"The separation of the DexYCB test set into 'seen' and 'unseen' categories is described as based on 'similarity in 3D shape' but no quantitative criterion is given. Please specify the similarity measure or provide a reference for the split.","section":"Section IV-D"},{"comment":"The paper does not report how many DexYCB frames or subjects are used in the reconstruction evaluation. Adding this information would help reproduce the results.","section":"Section IV-A"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of RA-L and the real-robot experiments are a positive aspect. The main concern is the input modality confound in the reconstruction comparison: if the reported gains over IHOI and SVHO partly stem from the use of segmentation masks, the headline claim of the abstract is not yet supported. This is fixable within the manuscript's scope by re-running the comparison with matched inputs or clearly repositioning the claim. The near-absence of uncertainty quantification is also a concern for a journal submission, but it is a presentation/analysis issue rather than a fundamental flaw. I believe major revision is appropriate rather than rejection, since the proposed combination of codebook priors and multi-view fusion is novel and the handover results are promising."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — short version: this is a genuine RGB-stereo hand-object reconstruction and handover system with real robot experiments, shipped code, and a plausible new fusion mechanism. But the central comparison is partly confounded: StereoHO feeds hand/object segmentation masks as input, and its own ablation shows masks improve object Chamfer distance. The headline gains over IHOI and SVHO are therefore not cleanly attributable to probabilistic stereo fusion. The stress-test note lands.\n\nWhat's actually new: per-view probability distributions over learned VQ-VAE shape codebooks, combined by element-wise product into a single reconstruction. That's a reasonable way to fuse two single-view hypotheses, and I don't know of prior stereo hand-object work doing it. Training on ObMan and evaluating on DexYCB plus real handovers means the shape-prior result isn't fitted to the test set. Code and videos are available, baselines are retrained on the same synthetic data, and the robot experiments are physical trials, not simulation. That is real evidence. The citation pattern also looks legitimate: the self-cited baselines are comparison points, not inputs.\n\nSoft spots, in proportion. First and most important, the mask confound: Section III-A says masks are inputs; Fig. 7 shows the masked version improves object CD while slightly hurting hand CD. The paper doesn't say whether DexYCB masks are ground truth or predicted, and the baselines don't get masks. So 'reduces object CD compared to RGB methods' is not apples-to-apples; an unknown part of the 9.71 and 35.43 cm^2 gaps may just be the extra input. Fixable by reporting masked/unmasked for all methods, but as written it muddies the main quantitative claim.\n\nSecond, no error bars or significance tests anywhere. The gaps look large and the conclusions would probably survive, but the paper doesn't show it.\n\nThird, the P=PLPR fusion assumes per-view codebook distributions are conditionally independent given the true shape. That's unvalidated; correlated errors could make product fusion miscalibrated. Comparing product against averaging or a learned fusion would settle it.\n\nFourth, the handover table compares StereoHO with CB under different initial conditions: CB starts with the object on the table, StereoHO with it already held. The 'on par' container result is weaker than it looks. Household-object numbers are encouraging but small-sample.\n\nWho benefits: people building RGB-based handover pipelines and stereo shape fusion. It deserves a serious referee. My recommendation: engage, but treat the headline CD claim as provisional until the mask-controlled comparison is run.","headline":"Real stereo RGB handover system with a genuine fusion idea, but the headline CD gains are partly confounded by segmentation-mask inputs; engage with the paper, not with its abstract.","tokens_in":13741,"tokens_out":3813,"would_cite":true,"duration_ms":38524,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"StereoHO reconstructs hand and object shape by multiplying two RGB views' probabilistic shape guesses, enough for a robot to receive transparent and thin household objects in handovers.","keywords":["hand-object reconstruction","stereo RGB","human-to-robot handover","shape codebook","signed distance field","transparent objects","vector quantization","6-DoF grasping"],"falsifier":"Use a test object visible from both cameras, introduce a shared error by offsetting both predicted wrist poses in the same direction, and compare the product-fused reconstruction with the better of the two single-view reconstructions; if fusion no longer beats the best single view, the independence assumption is falsified. A complementary check is to score the fused probability distribution against ground-truth SDF occupancy: an overconfident product would show systematically lower calibration than either view alone.","tokens_in":12777,"feed_emoji":"🤝","tokens_out":7103,"duration_ms":68888,"temperature":0.7,"pith_summary":"The paper tries to establish that a robot can receive a wide range of household objects from a human using only two ordinary RGB cameras, without depth sensors and without assuming anything about the object's shape. The key idea is to reconstruct the hand and object probabilistically: each camera view proposes a distribution over learned 3D shape patches, and the two views are combined by multiplying these distributions into one coherent shape. The authors train on synthetic data and test on real data, reporting lower object reconstruction error than prior RGB methods in both single-view and stereo settings. In physical handover experiments, the reconstructed shape lets a robot grasp and receive diverse objects, including transparent containers that defeat depth-based methods, with 75 percent grasping success on opaque household objects and 91 percent on transparent ones. This matters because transparent and oddly shaped objects are exactly the cases where depth cameras and hand-crafted shape priors fail.","feed_headline":"Two RGB views beat depth on transparent robot handovers","feed_subtitle":"Combining per-view shape-code probabilities lets a robot receive see-through and thin objects without depth sensors.","key_machinery":"The machinery is a learned shape codebook plus a product fusion rule. A shape codebook is a finite set of quantized 3D patch embeddings learned by a vector-quantized autoencoder from synthetic hand-object data. For each camera view, StereoHO predicts a probability distribution over codebook entries for every voxel in a grid centred on the wrist. The central identity is the fusion rule $P = P_L P_R$ (element-wise multiplication of the left and right per-voxel distributions), which yields a joint stereo distribution that is then decoded to a truncated signed distance field. The same product is computed independently for hand and object, and a projection-based outlier removal step enforces consistency with the observed segmentation masks.","core_discovery":"StereoHO jointly reconstructs hand and object geometry from wide-baseline stereo RGB images by treating shape prediction as classification over a learned codebook of 512 discrete 3D shape embeddings. For each view, a convolutional encoder produces a per-voxel probability distribution over this codebook for hand and object separately; the stereo prediction is the element-wise product of the left and right distributions, decoded by SDF decoders into a truncated signed distance field and then a pointcloud. A projection-based outlier removal step filters reconstructed points against segmentation masks in both views. On the DexYCB test set, the method reports object Chamfer distance improvements over IHOI in the single-view setting and over SVHO in the stereo setting, while remaining comparable on hand reconstruction. In the handover pipeline, this reconstruction supports 6-DoF grasp planning and hand-collision avoidance, achieving over 75 percent grasping success on CORSMAL containers, 75 percent on opaque household objects, and 91 percent on transparent containers, outperforming depth-based baselines on transparent objects.","pith_inferences":["Editorial inference: what is not tested in the paper is whether three or more views fused by the same product rule would keep reducing reconstruction error, which would confirm the independence assumption rather than just the two-view result.","Editorial inference: the per-voxel probabilities over codebooks could be read as calibrated confidence, letting the robot ask the human to reposition when the stereo views disagree or when the top codebook probability is low; the paper does not explore this.","Editorial inference: because the components are trained purely on synthetic data with domain-invariant masks, the same architecture is a candidate for other RGB-only perception tasks such as teleoperated grasping or assistive dressing, where transparent or deformable objects appear.","Editorial inference: the frame-by-frame reconstruction-quality gating suggested in the pipeline indicates that temporal fusion of the probability distributions across frames could further stabilise handovers; the paper instead replaces rather than accumulates predictions."],"forward_implications":["Ordinary stereo RGB cameras become sufficient for safe human-to-robot handover of unknown objects, including transparent and reflective ones, removing the depth-camera requirement.","The method transfers from synthetic training data to real, unseen household objects without hand-crafted geometric priors, so non-container objects and arbitrary grasp orientations are handled.","Multi-view coherence is achieved probabilistically at the shape-code level rather than by merging pointclouds, which naturally gives a per-voxel uncertainty signal.","Object reconstruction error drops when using two views instead of one, while hand reconstruction stays roughly at the same quality as single-view methods.","The reconstructed pointcloud can directly drive 6-DoF grasp estimation and hand-collision filtering in a closed-loop robot pipeline."],"supporting_citations":[{"why":"Supplies the synthetic ObMan hand-object dataset used to train the codebooks, SDF decoders, and image-to-shape encoder.","marker":"[13]"},{"why":"IHOI, the single-view SDF reconstruction method that StereoHO compares against and whose image-to-voxel formulation it adapts to stereo.","marker":"[20]"},{"why":"SVHO, the sparse multi-view method that serves as the stereo baseline for reconstruction comparison.","marker":"[27]"},{"why":"DexYCB, the multi-view real dataset used to evaluate single-view and stereo reconstruction quality.","marker":"[18]"},{"why":"AutoSDF, the patch-wise vector-quantized variational autoencoder approach used to learn the discrete shape codebooks.","marker":"[29]"},{"why":"Provides the vector-quantization mechanism that maps continuous shape codes to discrete codebook entries.","marker":"[30]"},{"why":"The CORSMAL benchmarking protocol that defines the handover setup, configurations, and performance measures.","marker":"[4]"},{"why":"The stereo container-specific handover baseline compared in the physical handover experiments.","marker":"[6]"},{"why":"6-DoF GraspNet, which generates candidate robot grasps from the reconstructed object pointcloud.","marker":"[15]"},{"why":"FrankMocap, which predicts the wrist pose and hand segmentation mask used by the handover pipeline.","marker":"[34]"}],"fun_headline_variants":["Stereo RGB outdoes depth on transparent robot handovers","No depth sensors: stereo RGB sees glass for robot handover","Probabilistic stereo shape codes beat depth on transparent items","Two-view shape codes fix transparent object handovers","Stereo RGB codebook outperforms depth on see-through objects"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The stereo fusion rule $P = P_L P_R$ assumes that, once the true hand-object shape is fixed, the two camera views' errors are conditionally independent, so multiplying their per-voxel codebook probabilities produces a calibrated joint probability rather than an overconfident one.","fun_headline_variants_meta":{"raw":{"variants":["Stereo RGB outdoes depth on transparent robot handovers","No depth sensors: stereo RGB sees glass for robot handover","Probabilistic stereo shape codes beat depth on transparent items","Two-view shape codes fix transparent object handovers","Stereo RGB codebook outperforms depth on see-through objects"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000683,"raw_usage":{"total_tokens":3090,"prompt_tokens":925,"completion_tokens":2165,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":2084}},"tokens_in":541,"tokens_out":2165,"duration_ms":16466,"temperature":1.0,"reasoning_tokens":2084,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:47:45.447238+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use a test object visible from both cameras, introduce a shared error by offsetting both predicted wrist poses in the same direction, and compare the product-fused reconstruction with the better of the two single-view reconstructions; if fusion no longer beats the best single view, the independence assumption is falsified. A complementary check is to score the fused probability distribution against ground-truth SDF occupancy: an overconfident product would show systematically lower calibration than either view alone.","supporting_citations":[{"cited_title":"Learning joint reconstruction of hands and manipulated objects,","cited_arxiv_id":null,"evidence_quote":"Supplies the synthetic ObMan hand-object dataset used to train the codebooks, SDF decoders, and image-to-shape encoder."},{"cited_title":"What’s in your hands? 3D recon- struction of generic objects in hands,","cited_arxiv_id":null,"evidence_quote":"IHOI, the single-view SDF reconstruction method that StereoHO compares against and whose image-to-voxel formulation it adapts to stereo."},{"cited_title":"Sparse multi-view hand-object reconstruction for unseen environments,","cited_arxiv_id":null,"evidence_quote":"SVHO, the sparse multi-view method that serves as the stereo baseline for reconstruction comparison."},{"cited_title":"DexYCB: A benchmark for capturing hand grasping of objects,","cited_arxiv_id":null,"evidence_quote":"DexYCB, the multi-view real dataset used to evaluate single-view and stereo reconstruction quality."},{"cited_title":"AutoSDF: Shape priors for 3D completion, reconstruction and generation,","cited_arxiv_id":null,"evidence_quote":"AutoSDF, the patch-wise vector-quantized variational autoencoder approach used to learn the discrete shape codebooks."},{"cited_title":"Neural discrete representation learning,","cited_arxiv_id":null,"evidence_quote":"Provides the vector-quantization mechanism that maps continuous shape codes to discrete codebook entries."},{"cited_title":"Benchmark for human-to-robot handovers of unseen containers with unknown filling,","cited_arxiv_id":null,"evidence_quote":"The CORSMAL benchmarking protocol that defines the handover setup, configurations, and performance measures."},{"cited_title":"Towards safe human- to-robot handovers of unknown containers,","cited_arxiv_id":null,"evidence_quote":"The stereo container-specific handover baseline compared in the physical handover experiments."},{"cited_title":"6-DoF GraspNet: Variational grasp generation for object manipulation,","cited_arxiv_id":null,"evidence_quote":"6-DoF GraspNet, which generates candidate robot grasps from the reconstructed object pointcloud."},{"cited_title":"FrankMocap: A monocular 3D whole-body pose estimation system via regression and integration,","cited_arxiv_id":null,"evidence_quote":"FrankMocap, which predicts the wrist pose and hand segmentation mask used by the handover pipeline."}],"review_version":1}