{"id":"32c322a4-9edc-4f8c-8807-349ba5a81981","arxiv_id":"2411.10013","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Stereo depth for AR glasses can skip rectification and replace cost volume with learned approximations, yielding lower latency and improved accuracy over Argos.","lead":"An AR glasses depth system replaces two slow standard steps, image rectification and cost volume search, with learned approximations: a multi-head dot-product cost and a homography-based positional encoding. The result is up to 44.5% lower end-to-end latency and better depth accuracy than the industrial Argos baseline on several datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"HomoDepth's single-homography RPE is theoretically invalid for near-field scenes; the central claim is untested on close-range AR content.","rationale":"The central novelty of HomoDepth is eliminating rectification preprocessing via a predicted homography and RPE. The geometric derivation in Sec. 4.1 explicitly assumes dl/dr ≈ 1, which is exact only for points on a central plane or at large distance. Stereo vision in general, and AR in particular, frequently contains close-range objects whose depth differs substantially from the dominant plane; for those points, a single homography cannot model the left-right correspondence. Since RPE is injected into the cost volume, incorrect positional encodings can degrade matching for near-field content. The paper's datasets do not isolate this regime: ADT excludes scenes with people (the most common near-field AR objects), DTU involves a single scanned object at moderate distances, and the perspective-transformed SceneFlow experiment reports global metrics only. Thus the accuracy and robustness claims for HomoDepth are not supported for a core AR use case. This is a load-bearing concern because it directly limits the scope of the paper's headline claim. The reader's weakest assumption correctly identifies this same issue. Other issues, such as the D1 metric inconsistency and missing hyperparameters, are addressable but do not strike at the core geometric validity of the approach. A depth-stratified evaluation on a near-field stereo benchmark or a synthetic scene with controlled depth ranges would settle whether the concern is practical or merely theoretical. Until then, the paper should be accepted conditionally, with the near-field limitation either explicitly acknowledged or resolved by additional experiments.","tokens_in":14629,"tokens_out":3460,"duration_ms":41494,"concrete_test":"Generate or select stereo images with a known foreground object at 0.2-0.5 m depth and background at >2 m, apply a known homography/perspective misalignment, run HomoDepth, and compute AbsRel separately for near and far depth bins. If the near-field AbsRel is more than 2x the far-field AbsRel while the baseline Argos with true rectification does not show this gap, the single-homography RPE fails in near-field AR scenes.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 4.1 derives the homography as an approximate rectification via Eq. (3), relying on dl/dr ≈ 1 for points on the central plane or far from the cameras. In AR, close objects such as hands, held items, and nearby furniture violate this condition. A single 3x3 homography cannot represent depth-dependent parallax, so RPE in Eq. (5) feeds incorrectly shifted positional encodings to all pixels of the right feature map. The paper's own robustness analysis (Appendix E) shows that depth estimation uses the predicted homography, and the ablation in Tab. 4 indicates RPE contributes to the gains; if the encoding is wrong for near-field pixels, it actively harms accuracy. The evaluation does not probe this regime: ADT scenarios containing people are excluded, DTU uses a single object at moderate range, and SceneFlow-persp applies a global perspective transform and reports only aggregate error. Consequently, the headline improvement for HomoDepth on unrectified images is established only for scenes where the planar approximation holds; the claim as stated is overbroad.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses latency bottlenecks in stereo depth estimation for AR glasses. It proposes MultiHeadDepth, which replaces the conventional cost volume with a multi-head group-pointwise-convolution operator plus a LayerNorm-dot-product approximation of cosine similarity, and HomoDepth, which adds a homography estimation head and a rectification positional encoding (RPE) so that unrectified stereo images can be processed without explicit rectification preprocessing. The models are evaluated on SceneFlow, ADT, DTU, and Middlebury against ARGOS, Dynamic-Stereo, MobileStereoNet, and Selective-Stereo, with latency measurements on a laptop CPU/GPU, Jetson Orin Nano, and Snapdragon. The authors report 11.8-30.3% accuracy improvements and 22.9-25.2% latency reduction for MultiHeadDepth over ARGOS, and a 44.5% end-to-end latency reduction for HomoDepth.","tokens_in":14777,"tokens_out":10715,"duration_ms":113060,"significance":"If the reported results are reproducible, the paper makes a useful engineering contribution: the cost-volume replacement is hardware-friendly and general, and the homography-plus-RPE formulation is a plausible way to avoid expensive rectification on unstable camera mounts. Strengths include the public code release, evaluation on a realistic AR dataset (ADT), latency measurements on three platforms including edge-scale hardware, and an explicit robustness analysis of homography noise in Appendix E. The evaluation is empirically self-contained and I do not see a circularity problem in the DTU homography ground truth, which is derived from extrinsic parameters rather than from the depth output. However, the headline claims are stronger than the evidence: the single-homography assumption is untested for near-field AR content, the D1 metric is incorrectly defined, and no variance or statistical analysis is provided. These issues need to be addressed before the specific quantitative claims can be accepted.","major_comments":[{"comment":"The central novelty of HomoDepth rests on a planar-scene assumption that is not tested. Section 4.1, Eq. (3) derives a single homography H_{l->r} under the approximation d_l/d_r ≈ 1, which holds only for points on the central plane of the cameras or far from the cameras. Section 4.3 then applies RPE(q_r) = PE(H_{l->r} q_l) to every pixel of the right feature map. For near-field objects such as hands, held items, and nearby furniture, a single 3x3 homography cannot represent depth-dependent parallax, so the RPE encodes a systematically wrong positional relationship for those pixels, and because the RPE is added inside the matching cost, the error can actively harm the learned similarity rather than acting as a benign input warp. The evaluation does not probe this regime: ADT scenarios with people are excluded (Section 5.1), DTU contains a single object at moderate range, Sceneflow-persp is a global perspective transform, and Appendix E studies homography noise but not scene-depth composition. The claim that HomoDepth can directly process unrectified images should either be tested on close-range content or explicitly scoped to scenes where the planar approximation holds.","section":"4.1, 4.3, 5.1, Table 4, Appendix E"},{"comment":"The D1 metric is defined as an inlier rate, D1 = (1/N) sum 1(|y_hat - y| / y <= 5%), which is higher-is-better, but the paper consistently treats D1 as lower-is-better and uses it in the claimed accuracy improvements. If D1 is meant to be a bad-pixel ratio, the inequality should be reversed or the complement taken. As written, the D1 columns in Tables 2 and 4 are internally inconsistent with the text: for example, on SceneFlow the MultiHeadDepth entry (0.35) is larger than the ARGOS entry (0.23), which contradicts the reported improvement if D1 is an error rate, and contradicts the stated 'lower is better' convention if D1 is an inlier rate. This is not a cosmetic issue because D1 is part of the quantitative accuracy claims.","section":"Appendix D, Eq. (14), Tables 2 and 4"},{"comment":"All accuracy and latency numbers are reported as single point estimates with no error bars, no multiple seeds, and no repeated hardware measurements. Section 5.2 says the authors 'select the best epoch,' and if this selection is done on the test set it can inflate apparent gains; even if done on validation, the absence of seed variance makes the smaller differences hard to interpret (e.g., AbsRel 0.102 vs 0.091 on SceneFlow in Table 2). Latency measurements on hardware also vary across runs and should be reported with means and ranges. I request variance over at least three training seeds and multiple latency trials so that the claimed 11.8-30.3% accuracy improvements can be assessed.","section":"5.2, 5.3, Tables 2-5"},{"comment":"The edge-device latency table does not support the end-to-end claim on those platforms. Table 5 reports model-only latencies for ARGOS and notes that preprocessing delay is excluded, while HomoDepth's advertised advantage is end-to-end latency. On Snapdragon CPU, HomoDepth is actually slower than ARGOS in the reported numbers (1512 ms vs 1424 ms), and on Jetson Orin Nano CPU it is slower than MultiHeadDepth (6611 ms vs 6183 ms). Without also measuring the preprocessing cost that HomoDepth removes on the same platforms, the general claim that HomoDepth reduces end-to-end latency is only demonstrated for the laptop configuration in Table 4. Please report end-to-end measurements on the edge devices, or clearly scope the latency claim to configurations where preprocessing costs were included.","section":"5.6, Table 5"}],"minor_comments":[{"comment":"The section title contains a typo: 'Postional' should be 'Positional.'","section":"4.3"},{"comment":"The exponents in the positional encoding formula are ambiguous in the typeset text; they should appear as f^{i/d} and f^{(i-1)/d} to match the standard sinusoidal encoding.","section":"Eq. (4)"},{"comment":"The loop 'for h in 0 : head num' should iterate over 0 to head_num-1; as written, h = head_num would index outside the channel slices.","section":"Algorithm 1"},{"comment":"The caption reads 'whose size is 1 x number of channels number of heads'; it should say 'number of channels divided by number of heads.'","section":"Fig. 3 caption"},{"comment":"The claimed latency reduction ranges (22.9-25.2% and 44.5%) should specify the exact comparison basis: model-only latency, model plus rectification, or full end-to-end including calibration. The current text uses different bases in different sections, which makes the headline numbers hard to reproduce from Tables 4 and 5.","section":"Abstract and Section 5.4"}],"recommendation":"major_revision","confidential_remarks":"To the editor: I believe this paper is a candidate for revision rather than rejection. The cost-volume replacement is a solid engineering contribution and the public code release is valuable. The most consequential gap is the untested near-field validity of the single-homography RPE, which is load-bearing for HomoDepth's central claim. The D1 metric definition error must also be corrected because it affects the accuracy comparisons. I would ask the authors for close-range experiments, variance reporting, and a consistent latency basis before reconsidering the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a read if you care about efficient stereo depth on wearables. The two ideas—replacing the cost volume with layer-norm-plus-dot-product and group-pointwise convolution, and encoding a predicted homography as a positional adjustment to avoid explicit rectification—are both practical and, in combination, new. The evaluation is more thorough than most systems papers in this space: three datasets plus Middlebury, three hardware platforms, quantized models, and an ablation that separates the rectification and cost-volume contributions. The latency numbers hold together directionally, and the accuracy improvements over Argos are consistent.\n\nThe soft spots are real but mostly fixable. The D1 metric in Appendix D is defined as the fraction of pixels within 5% error, which would make it a higher-is-better accuracy, yet the text and tables treat it as lower-is-better; the numbers only make sense if it is actually a bad-pixel rate with a '>' in Eq. (14). That has to be corrected. There are no error bars or repeated runs, so the 11.8–30.3% accuracy range is a single-run point estimate. The abstract's 44.5% end-to-end latency reduction for HomoDepth is hard to reproduce from Table 4—the comparison baseline is not clear.\n\nThe deeper concern, which the stress-test flags correctly, is that HomoDepth rests on a single homography approximation that the paper itself admits breaks down when dl/dr is not close to 1. That is exactly the near-field regime in AR: hands, held objects, nearby furniture. The evaluation excludes humans from ADT, uses DTU at moderate range, and applies a global perspective transform to SceneFlow. So the claim that HomoDepth 'directly processes unrectified images' is established only for scenes where the planar approximation holds, not for the close-range content that matters in AR. The paper should either test near-field scenes or qualify the claim.\n\nThat said, the central engineering contribution—the cost volume replacement—is solid and does not depend on the homography approximation. I would send this to reviewers. It is a systems paper, not a theoretical one, but the problem is important and the approach is coherent. The authors should be asked to fix the D1 definition, report repeated-run statistics and missing hyperparameters, and add a near-field evaluation or narrow the claim.","headline":"Solid systems paper on replacing cost volume and rectification for stereo depth on AR glasses; the HomoDepth homography approximation is untested on near-field scenes and the D1 metric definition is inconsistent, but the core latency/accuracy claims are directionally plausible and worth refereeing.","tokens_in":15338,"tokens_out":2645,"would_cite":true,"duration_ms":27402,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A stereo depth network for AR glasses replaces geometric rectification and the traditional cost volume with learned, hardware-friendly operations, cutting end-to-end latency by 44.5% while improving accuracy.","keywords":["stereo depth estimation","AR glasses","homography estimation","cost volume","rectification positional encoding","multi-task learning","latency optimization","layer normalization"],"falsifier":"Run HomoDepth on a fixed unrectified stereo rig while a hand or object moves from about two meters to 20 cm from the cameras, and compare AbsRel against the same model given properly rectified images: if the error gap grows sharply in the near field while far-background accuracy stays flat, the global homography assumption is the cause. A simpler version is to replace the predicted single homography with per-pixel or piecewise homographies and observe whether near-field errors fall.","tokens_in":14343,"feed_emoji":"🥽","tokens_out":7138,"duration_ms":71740,"temperature":0.7,"pith_summary":"The paper tries to establish that the two most latency-heavy non-machine-learning steps in stereo depth estimation on AR glasses—image rectification and the cost volume—can be replaced by learned operators that run well on GPUs and NPUs. Its MultiHeadDepth model swaps the cost volume for a layer-normalized, group-wise dot-product block with pointwise convolution, and its HomoDepth model adds a homography-estimation head whose output becomes a 2D rectification positional encoding, so unrectified stereo pairs can go straight into the network. The reported result is that MultiHeadDepth improves accuracy by 11.8-30.3% and reduces cost-volume-plus-preprocessing latency by 22.9-25.2% relative to the ARGOS baseline, and HomoDepth cuts end-to-end latency by 44.5%. If these results hold, AR depth estimation could avoid online calibration and rectification entirely, which matters because glasses frames bend and extrinsic parameters drift during use.","feed_headline":"AR depth estimation drops rectification and cuts latency 44.5%","feed_subtitle":"A predicted homography plus positional encoding lets stereo models accept unrectified images directly, with accuracy gains too.","key_machinery":"The carrying mechanism is the rectification positional encoding (RPE), a 2D sinusoidal positional encoding evaluated at rectified coordinates: pixels on the left use $\\mathrm{PE}(q_l)$, pixels on the right use $\\mathrm{PE}(H_{l\\to r}q_l)$, so a world point produces similar encodings in both views and the matching layers can compare appearance and position even when the raw images are unrectified. It is enabled by a homography-estimation head sharing the encoder with the depth head. The second mechanism is the multi-head cost volume: layer normalization followed by per-disparity rolling, group-wise dot products, and pointwise convolution, which replaces cosine similarity with matrix-multiplication-friendly operators. The two are trained jointly with a homoscedastic-uncertainty loss so that homography error and depth error are weighed automatically.","core_discovery":"The central claim is that rectification preprocessing can be represented, not executed: instead of resampling the right image to make epipolar lines horizontal, the network predicts the 3x3 homography from the shared encoder, converts it into a rectification positional encoding (RPE) $\\mathrm{RPE}(q_r) := \\mathrm{PE}(H_{l\\to r}q_l)$, and adds that encoding inside the cost volume so matching can use positional alignment without image warping. The traditional cost volume is replaced by a multi-head formulation that layer-normalizes both feature maps, rolls one map by each disparity, computes group-wise dot products, and fuses them with a pointwise convolution. This acts as a hardware-friendly approximation of cosine similarity while adding learnable weights and multi-head perception. On the ARGOS architecture, the authors report that MultiHeadDepth delivers 11.8-30.3% accuracy improvement and 22.9-25.2% latency reduction over ARGOS, and HomoDepth accepts unrectified images directly with a 44.5% end-to-end latency reduction, with the multi-task homoscedasticity-weighted loss cutting AbsRel error by 10.0-24.3% on misaligned stereo inputs.","pith_inferences":["Editor's inference: the single-homography assumption should be stress-tested on near-field content; hands, held objects, and close furniture violate $d_l/d_r\\approx 1$, so the RPE would encode wrong geometry exactly where AR occlusion and manipulation matter.","Editor's inference: the group-pointwise cost volume is not specific to depth; optical flow, stereo matching for robotics, and multi-view stereo could swap the same operator in wherever cosine-similarity cost volumes dominate latency.","Editor's inference: because RPE is generated from a homography at inference time, the architecture could be extended to predict a piecewise or depth-dependent transformation (e.g., multiple homographies for foreground/background) rather than one global planar map, and the paper's robustness curves suggest the depth head tolerates small homography noise but not large errors.","Editor's inference: a direct comparison against classical rectification plus matching on a close-range AR interaction benchmark (hands, held objects within arm's reach) would be the decisive test of whether preprocessing can be fully eliminated in practice."],"forward_implications":["Any stereo depth model built on a cost volume can adopt the multi-head cost volume without changing its encoder-decoder, and should inherit most of the reported latency reduction on matrix-multiplication hardware.","AR glasses whose frames bend or whose extrinsics drift no longer need an explicit online rectification step; HomoDepth consumes the raw unrectified pair directly, removing the 15-23% rectification failure rate reported for the fast online rectification baseline.","End-to-end one-shot depth estimation can fit inside the 100 ms real-time budget on laptop-class GPU hardware (84.5 ms for HomoDepth versus 109.0 ms for rectified ARGOS in the paper's measurements).","Because the homography head shares the encoder, continuous operation can reuse a stable homography across nearby frames and run only the depth path, cutting per-frame cost further.","Quantization to INT8 preserves the accuracy and latency advantages, suggesting the operators survive post-training quantization needed for on-device deployment."],"supporting_citations":[{"why":"Provides the ARGOS baseline whose preprocessing and cost-volume latency are analyzed and whose cost volume is replaced.","marker":"[31]"},{"why":"Supplies the convolutional homography-estimation approach that the homography head adapts.","marker":"[5]"},{"why":"Demonstrates homography-based depth reasoning with known camera parameters, the stricter setting this paper relaxes.","marker":"[33]"},{"why":"Provides the homoscedastic uncertainty loss used to train the two heads jointly.","marker":"[15]"},{"why":"Supplies SceneFlow, the synthetic rectified training set used for stereo matching and the similarity map in the paper's comparison.","marker":"[21]"},{"why":"Supplies ADT, the Aria AR-glasses dataset used to evaluate performance on realistic egocentric images.","marker":"[22]"},{"why":"Supplies DTU, the unrectified dataset with known extrinsics used to create homography ground truth and test preprocessing-free depth.","marker":"[12]"},{"why":"Provides the 2D positional encoding formula that RPE adapts to the rectified coordinate system.","marker":"[29]"}],"fun_headline_variants":["AR depth without rectification: predicted homography cuts latency 44.5%","Skip stereo warping: learn homography, add positional encoding","Faster AR depth: replace rectification and cost volume","HomoDepth: 44.5% latency cut by predicting rectification","MultiHeadDepth: up to 30% accuracy gain, 25% latency cut"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a single $3\\times 3$ homography can represent the left-right relationship of every pixel in the scene, which Eq. (3) makes exact only when $d_l/d_r\\approx 1$, i.e., for points on the central plane or far away; close objects such as hands or held items in AR scenes break this condition and would receive a systematically wrong rectification encoding.","fun_headline_variants_meta":{"raw":{"variants":["AR depth without rectification: predicted homography cuts latency 44.5%","Skip stereo warping: learn homography, add positional encoding","Faster AR depth: replace rectification and cost volume","HomoDepth: 44.5% latency cut by predicting rectification","MultiHeadDepth: up to 30% accuracy gain, 25% latency cut"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000484,"raw_usage":{"total_tokens":2469,"prompt_tokens":1107,"completion_tokens":1362,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":723,"completion_tokens_details":{"reasoning_tokens":1265}},"tokens_in":723,"tokens_out":1362,"duration_ms":12147,"temperature":1.0,"reasoning_tokens":1265,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:04:32.728432+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run HomoDepth on a fixed unrectified stereo rig while a hand or object moves from about two meters to 20 cm from the cameras, and compare AbsRel against the same model given properly rectified images: if the error gap grows sharply in the near field while far-background accuracy stays flat, the global homography assumption is the cause. A simpler version is to replace the predicted single homography with per-pixel or piecewise homographies and observe whether near-field errors fall.","supporting_citations":[{"cited_title":"A practical stereo depth system for smart glasses","cited_arxiv_id":null,"evidence_quote":"Provides the ARGOS baseline whose preprocessing and cost-volume latency are analyzed and whose cost volume is replaced."},{"cited_title":"Deep image homography estimation, 2016","cited_arxiv_id":null,"evidence_quote":"Supplies the convolutional homography-estimation approach that the homography head adapts."},{"cited_title":"Mvsnet: Depth inference for unstructured multi-view stereo","cited_arxiv_id":null,"evidence_quote":"Demonstrates homography-based depth reasoning with known camera parameters, the stricter setting this paper relaxes."},{"cited_title":"Aria digital twin: A new benchmark dataset for egocentric 3d machine percep- tion","cited_arxiv_id":null,"evidence_quote":"Supplies ADT, the Aria AR-glasses dataset used to evaluate performance on realistic egocentric images."},{"cited_title":"Large scale multi-view stereopsis eval- uation","cited_arxiv_id":null,"evidence_quote":"Supplies DTU, the unrectified dataset with known extrinsics used to create homography ground truth and test preprocessing-free depth."},{"cited_title":"Attention is all you need","cited_arxiv_id":null,"evidence_quote":"Provides the 2D positional encoding formula that RPE adapts to the rectified coordinate system."}],"review_version":1}