REVIEW 4 major objections 6 minor 22 references
Learning Robust Stereo Matching in the Wild with Selective Mixture-of-Experts
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read By inserting rank- and kernel-varying experts into frozen vision foundation models, SMoEStereo claims state-of-the-art zero-shot stereo matching across KITTI, Middlebury, ETH3D, and DrivingStereo with one fixed model.
desk verdict Solid empirical PEFT-MoE framework for robust stereo; architecture and ablations are convincing, but SOTA claims need error bars and working code. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing objects are the MoE-LoRA layer, whose experts are LoRA matrices with distinct ranks $r_i$ selected by a softmax router, and the MoE-Adapter layer, whose experts are $1\times1$ to $k\times k$ CNN blocks with different kernels selected the same way. A decision network reads the class token, uses a Gumbel-softmax relaxation to produce binary masks $\mathcal{M}^l_L$ and $\mathcal{M}^l_A$ for each layer, and a usage loss $\mathcal{L}_{\text{usage}}$ with budget $\gamma$ controls how many MoE layers stay active. This machinery converts 'which adaptation subspace fits this scene' into trainable per-input routing decisions and aligns computation with input complexity.
What would settle it
Train the model, then evaluate it on far out-of-distribution stereo pairs such as heavy rain, night, or specular and transparent surfaces, and record the decision network's per-layer retention masks; if the masks are statistically indistinguishable from random pruning at the same compute budget, the scene-conditional routing claim is falsified.
Extended reading notes
Core claim
The paper's central claim is that scene-specific, heterogeneous Mixture-of-Experts adaptation unlocks the robustness of vision foundation models for stereo matching. MoE-LoRA replaces fixed-rank LoRA with multiple rank choices routed per input, MoE-Adapter injects local inductive bias through CNN adapters of different kernel sizes, and a decision network trained with a usage loss learns binary per-layer skip policies. Together these components reduce D1 error by up to thirty percent versus a vanilla VFM-LoRA baseline while using fewer activated parameters at inference. The authors report state-of-the-art cross-domain generalization on KITTI, Middlebury, ETH3D, and DrivingStereo, and state-of-the-art joint generalization on the robust vision benchmarks with a single fixed model.
Load-bearing premise
The decision network decides which MoE layers to keep using only the class token of the image; if that single vector cannot reliably signal which layers a scene needs, the claimed efficiency gains and part of the accuracy gains would disappear.
Editorial extensions
If this is right
- A frozen vision foundation model backbone can reach top cross-domain stereo accuracy with only a few million trainable parameters, avoiding costly full fine-tuning.
- The same trained model can be deployed on indoor, outdoor, and low-resolution benchmarks without dataset-specific adaptation, which is the setting real-world stereo systems face.
- Varying the budget $\gamma$ yields a smooth accuracy-versus-speed trade-off, so one model can serve devices with different compute limits.
- Because the MoE blocks are plug-and-play, classic stereo backbones such as PSMNet, CFNet, and IGEV also gain in zero-shot transfer when wrapped in this design.
- Larger vision foundation model backbones amplify the gains, so the approach improves as stronger foundation models become available.
Reading between the lines
- One implication not pursued in the paper: the learned expert-selection distributions across datasets (KITTI, Middlebury, ETH3D) could act as a cheap domain fingerprint, suggesting the router encodes scene type rather than just difficulty.
- The class-token-only decision network is a minimal-routing bet; a multi-token or patch-level policy might generalize further, but would trade away the efficiency the paper is after.
- The same heterogeneous-expert parameter-efficient fine-tuning recipe could transfer to other dense prediction tasks that suffer from domain shift, such as optical flow or monocular depth, wherever a frozen vision foundation model is the starting point.
- The thirty percent D1 reduction relative to vanilla LoRA is measured on real benchmarks; a natural stress test is whether the routing policy holds on far out-of-distribution scenes such as night, snow, or specular surfaces.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SMoEStereo, a parameter-efficient fine-tuning framework that adapts frozen Vision Foundation Models to stereo matching via two mixture-of-experts modules: MoE-LoRA with experts of different ranks and MoE-Adapter with experts of different kernel sizes, plus a lightweight decision network that predicts binary per-layer usage policies to selectively activate these modules. The method is evaluated for cross-domain generalization (pretraining on SceneFlow, testing on KITTI, Middlebury, ETH3D, DrivingStereo) and joint generalization (RVC setting, one model evaluated on the KITTI 2015, Middlebury, and ETH3D benchmarks). The paper reports state-of-the-art or near-state-of-the-art results, fewer trainable parameters than full fine-tuning, and improved inference efficiency compared with prior VFM-based stereo approaches. Ablations in the main text and supplementary show each component contributes, and the learned routing policy outperforms random policies of similar compute.
Significance. The paper is timely: leveraging VFMs for dense matching with a parameter-efficient, plug-and-play scheme is of broad interest. The heterogeneous MoE design (different LoRA ranks and CNN kernel sizes) with a trainable layer-selection mechanism is a technically interesting contribution, and the reported gains over the strongest VFM-based baseline (Former-RAFT) and over other PEFT methods are notable if they hold. The experimental coverage is extensive: multiple VFM backbones, PEFT baselines, cross-domain and RVC benchmarks, efficiency comparisons, and ablations. The method also ships a public code link, which aids reproducibility. The main fragility is that the headline 'state-of-the-art' claims rely on single-seed results with small margins and without significance tests, and one abstract claim is not supported by the paper's own Table 1.
major comments (4)
- [Abstract and Table 1] The abstract claims 'state-of-the-art cross-domain generalization performance on the KITTI, Middlebury, ETH3D, and DrivingStereo datasets.' However, Table 1 shows that Former-RAFT achieves 3.9 on KITTI 2012 Bad 3.0, while SMoEStereo (DAMV2) achieves 4.22, and SMoEStereo (DINOv2) achieves 4.39. Thus the unqualified 'state-of-the-art on KITTI' claim is contradicted by the paper's own numbers. The claim should either be qualified as 'state-of-the-art on most benchmarks' or the paper should explain why KITTI 2012 is an exception and why the overall conclusion still holds.
- [Tables 1, 2, 6] No error bars, multiple seeds, or significance tests are reported anywhere. Several margins that anchor the state-of-the-art claims are very small: in Table 6, the ETH3D AvgErr is 0.14 for both SMoEStereo and LoS, and the Middlebury bad 2.0 is worse for SMoEStereo (9.74) than for LoS (9.30); in Table 1, KITTI 2015 Bad 3.0 differs by 0.03 between SMoEStereo (DAMV2) and Former-RAFT; in Table 2, DrivingStereo average D1 differs by 0.7 from Former-CFNet. With a single seed, these differences could easily be within run-to-run noise. The paper should report at least 3 seeds with means and standard deviations (or equivalent), and for the 'overall rank' claim in Table 6, it should state whether the rank is stable across runs. This is load-bearing because the 'state-of-the-art' and 'significantly outperforming' statements in Sec. 4.2.2 rest on these fine-grained margins.
- [Sec. 3.2, Eq. (2)] The dimensionality of the LoRA expert matrices is inconsistent. The paper writes W_up^i ∈ R^{d×dim} and W_down^i ∈ R^{ri×ri}, but to compose as E_i^L(x_in) = W_up W_down x_in with x_in ∈ R^{N×dim}, W_down must map dim to ri (e.g., R^{dim×ri}) and W_up must map ri back to dim (R^{dim×ri} or R^{d×ri} with d=dim). As written, the product is undefined. This is a reproducibility issue in the core method formulation and should be corrected, along with clarifying the undefined quantity D in 'ri ≪ {D, dim}'.
- [Sec. 3.2, Decision Network; Tables 7 and 9] The decision network's core assumption is that the class token alone carries enough information to predict which MoE layers should be activated for a given input. This is validated only indirectly by the ablations in Table 7 (ID 5 vs. ID 6) and Fig. 5. The paper should provide a more direct analysis: e.g., report the routing policy's behavior on far-out-of-distribution inputs (such as DrivingStereo foggy/rainy subsets in the cross-domain setting), show examples where the policy skips critical layers and degrades accuracy, or quantify the accuracy/efficiency trade-off under mismatched routing. Without this, the efficiency claim and part of the accuracy claim rest on the unverified assumption that the learned policies generalize beyond the training domains.
minor comments (6)
- [Sec. 3.2, Eq. (7)] The notation x_cls ∈ R^{N×dim} for the class token is likely a typo; the class token is a single token, so x_cls ∈ R^{1×dim}. The description of using the first element M^l_1 as the binary probability is also confusing and should be clarified.
- [Table 9] The text states 'SMoE outperforms Multi-E in cross-dataset performance,' but on KITTI 2015 EPE Multi-E is 0.59 and SMoE is 0.60. This should be acknowledged or the claim qualified, since it is not uniformly true.
- [Tables and formatting] The paper claims 'sub-optimal best results in blue,' but the tables are not shown in color in the provided version, making this annotation invisible. Also, Table 5 names the GPU as 'Nvidia 5000 Ada' while the text says 'RTX 5000 Ada'; please unify.
- [Code availability] The code link in the abstract is given in red text and is not verifiable from the manuscript. Please ensure the link is operational in the final version and consider adding a reproducibility statement describing how the code can be used to reproduce the main tables.
- [Supplementary Table 10] The row '√ - -' has a stray quote character in the Middlebury column ('7.05), likely a LaTeX artifact. Please proofread the supplementary tables.
- [Conclusion] The paper does not include a limitations section. Given the single-seed evaluation and the unverified code link, adding a short limitations paragraph would improve the manuscript's transparency.
Circularity Check
No significant circularity: benchmark results are external and the routing mechanism is validated by a random-policy control.
full rationale
SMoEStereo's derivation chain is: define MoE-LoRA and MoE-Adapter modules, train on SceneFlow or on the RVC training splits, and evaluate on public KITTI/Middlebury/ETH3D/DrivingStereo benchmarks. No equation or fitted hyperparameter uses the test labels being predicted, so the headline cross-domain and joint-generalization results are not self-fulfilling. The decision-network claim is the component that could in principle be self-justifying, but Table 7 (ID 5 vs ID 6) and Fig. 5 compare learned policies against random policies of matched compute; the observed drop for random policies is an external control, not a by-construction guarantee. The paper does cite the overlapping prior work [79] as the closest related method, but that citation is used as a baseline for comparison, not as authority for a load-bearing assumption or any uniqueness claim. The absence of error bars and significance tests weakens the precision of the state-of-the-art claims, but that is an evidential gap, not circularity. No circular step is identifiable in the paper's derivations.
Assumptions & free parameters
free parameters (7)
- LoRA rank set =
{4, 8, 16, 32}
- Adapter kernel set =
{3, 5, 7, 9}
- Number of experts M and N =
M=4, N=4 (implied)
- Router temperature tau =
5
- Usage loss budget gamma =
not fixed; adjustable in (0,1]
- Loss weights lambda1, lambda2 =
1.0
- Disparity loss exponential weight beta =
0.9
assumptions (4)
- domain assumption Vision foundation models pretrained on large diverse image datasets provide features that transfer to stereo matching and can be adapted with small PEFT modules.
- domain assumption RAFT-Stereo's iterative cost aggregation is a suitable decoder for VFM-extracted features.
- ad hoc to paper The class token carries enough information to predict which MoE layers should be activated for a given input.
- standard math Gumbel softmax provides a valid differentiable relaxation for the binary layer masks.
Cite this review
Pith. "Pith review of Learning Robust Stereo Matching in the Wild with Selective Mixture-of-Experts." pith.science (2026). https://pith.science/paper/4XWLIP2X
@misc{pith2026250704631,
author = {Pith},
title = {Pith review of: Learning Robust Stereo Matching in the Wild with Selective Mixture-of-Experts},
year = {2026},
howpublished = {\url{https://pith.science/paper/4XWLIP2X}},
note = {Machine review of arXiv:2507.04631}
}
read the original abstract
Recently, learning-based stereo matching networks have advanced significantly. However, they often lack robustness and struggle to achieve impressive cross-domain performance due to domain shifts and imbalanced disparity distributions among diverse datasets. Leveraging Vision Foundation Models (VFMs) can intuitively enhance the model's robustness, but integrating such a model into stereo matching cost-effectively to fully realize their robustness remains a key challenge. To address this, we propose SMoEStereo, a novel framework that adapts VFMs for stereo matching through a tailored, scene-specific fusion of Low-Rank Adaptation (LoRA) and Mixture-of-Experts (MoE) modules. SMoEStereo introduces MoE-LoRA with adaptive ranks and MoE-Adapter with adaptive kernel sizes. The former dynamically selects optimal experts within MoE to adapt varying scenes across domains, while the latter injects inductive bias into frozen VFMs to improve geometric feature extraction. Importantly, to mitigate computational overhead, we further propose a lightweight decision network that selectively activates MoE modules based on input complexity, balancing efficiency with accuracy. Extensive experiments demonstrate that our method exhibits state-of-the-art cross-domain and joint generalization across multiple benchmarks without dataset-specific adaptation. The code is available at \textcolor{red}{https://github.com/cocowy1/SMoE-Stereo}.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Y ohann Cabon, Naila Murray, and Martin Humenberger. Vir- tual kitti 2. arXiv preprint arXiv:2001.10773, 2020. 6
arXiv 2001
-
[2]
Pyramid stereo matching network
Jia-Ren Chang and Y ong-Sheng Chen. Pyramid stereo matching network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5410–5418, 2018. 2, 4
work page 2018
-
[3]
Mocha-stereo: Motif channel attention network for stereo matching
Ziyang Chen, Wei Long, He Y ao, Y ongjun Zhang, Bing- shu Wang, Y ongbin Qin, and Jia Wu. Mocha-stereo: Motif channel attention network for stereo matching. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 27768–27777, 2024. 1
work page 2024
-
[4]
Lora: Low-rank adaptation of large language models
Edward J Hu, Y elong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Y uanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 3
arXiv 2021
-
[5]
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Y en Lo, et al. Segment any- thing. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 4015–4026, 2023. 1, 3
work page 2023
-
[6]
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Ui- jlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. Inter- national Journal of Computer Vision (IJCV) , 128(7):1956– 1981, 2020. 1
work page 1956
-
[7]
Pseudo-label: The simple and effi- cient semi-supervised learning method for deep neural net- works
Dong-Hyun Lee et al. Pseudo-label: The simple and effi- cient semi-supervised learning method for deep neural net- works. In Workshop on challenges in representation learn- ing, ICML, page 896. Atlanta, 2013. 1
work page 2013
-
[8]
Practical stereo matching via cascaded recurrent net- work with adaptive correlation
Jiankun Li, Peisen Wang, Pengfei Xiong, Tao Cai, Ziwei Y an, Lei Y ang, Jiangyu Liu, Haoqiang Fan, and Shuaicheng Liu. Practical stereo matching via cascaded recurrent net- work with adaptive correlation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 16263–16272, 2022. 6
work page 2022
Show all 22 references
-
[9]
RAFT-Stereo: Multilevel recurrent field transforms for stereo matching
Lahav Lipson, Zachary Teed, and Jia Deng. RAFT-Stereo: Multilevel recurrent field transforms for stereo matching. 2021 International Conference on 3D Vision (3DV) , pages 218–227, 2021. 1, 2
2021
-
[10]
A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation
Nikolaus Mayer, Eddy Ilg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Proceedings of the IEEE Conference on Computer Vision and Pa...
2016
-
[11]
Maxime Oquab, Timoth ´ee Darcet, Th´eo Moutakanni, Huy V . V o, Marc Szafraniec, V asil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Y ao Huang, Shang-Wen Li, Ishan Misra, Michae...
2024
-
[12]
Vi- sion transformers for dense prediction
Ren ´e Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vi- sion transformers for dense prediction. ArXiv preprint, 2021. 1
2021
-
[13]
High-resolution stereo datasets with subpixel-accurate ground truth
Daniel Scharstein, Heiko Hirschm ¨uller, Y ork Kitajima, Greg Krathwohl, Nera Ne ˇsi´c, Xi Wang, and Porter West- ling. High-resolution stereo datasets with subpixel-accurate ground truth. In German conference on pattern recognition (GCPR), pages 31–42. Springer, 2014. 6
2014
-
[14]
Outra- geously large neural networks: The sparsely-gated mixture- of-experts layer
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outra- geously large neural networks: The sparsely-gated mixture- of-experts layer. International Conference on Learning Rep- resentations (ICLR), 2017. 4
2017
-
[15]
CFNet: Cascade and fused cost volume for robust stereo matching
Zhelun Shen, Y uchao Dai, and Zhibo Rao. CFNet: Cascade and fused cost volume for robust stereo matching. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 13906–13915, 2021. 3, 4, 6
2021
-
[16]
Selective-stereo: Adaptive frequency information selection for stereo matching
Xianqi Wang, Gangwei Xu, Hao Jia, and Xin Y ang. Selective-stereo: Adaptive frequency information selection for stereo matching. arXiv preprint arXiv:2403.00486, 2024. 1, 2
2024 arXiv
-
[17]
Iterative geometry encoding volume for stereo matching
Gangwei Xu, Xianqi Wang, Xiaohuan Ding, and Xin Y ang. Iterative geometry encoding volume for stereo matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 21919–21928, 2023. 2, 3, 4, 6
2023
-
[18]
Depth any- thing v2
Lihe Y ang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiao- gang Xu, Jiashi Feng, and Hengshuang Zhao. Depth any- thing v2. arXiv preprint arXiv:2406.09414, 2024. 3, 4
2024 arXiv
-
[19]
Bdd100k: A diverse driving dataset for heterogeneous multitask learning
Fisher Y u, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, V ashisht Madhavan, and Trevor Dar- rell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVP...
2020
-
[20]
Robust synthetic-to-real transfer for stereo matching
Jiawei Zhang, Jiahe Li, Lei Huang, Xiaohan Y u, Lin Gu, Jin Zheng, and Xiao Bai. Robust synthetic-to-real transfer for stereo matching. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR) , pages 20247–20257, 2024. 2, 4
2024
-
[21]
Learning representations from foundation models for domain generalized stereo matching
Y ongjian Zhang, Longguang Wang, Kunhong Li, Y un Wang, and Y ulan Guo. Learning representations from foundation models for domain generalized stereo matching. In Euro- pean Conference on Computer Vision (ECCV) , pages 146–
-
[22]
High-frequency stereo match- ing network
Haoliang Zhao, Huizhou Zhou, Y ongjun Zhang, Jie Chen, Yitong Y ang, and Y ong Zhao. High-frequency stereo match- ing network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1327–1336, 2023. 1, 2
2023
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.