{"id":"c4fabea1-13ff-49bf-9a03-23c3b3ab46d3","arxiv_id":"2412.09912","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"AIO-Stereo transfers and selectively fuses knowledge from DINOv2, SAM, and Depth Anything into a stereo matching network, achieving top results on Middlebury and ETH3D.","lead":"This paper introduces AIO-Stereo, a stereo matching model that selectively distills knowledge from three vision foundation models (DINOv2, SAM, Depth Anything v2) into a single CNN-based stereo network. It reports state-of-the-art accuracy on the Middlebury and ETH3D benchmarks, which is relevant for robotics and autonomous driving applications.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Zero-shot gains are attributed to VFM distillation, but the only ablation isolating distillation is evaluated on the training set; if it is not run zero-shot, the central generalization claim is untested.","rationale":"I agree with the reader's weakest assumption: the decisive question is whether the learned alignment and gating transfer generic knowledge or just fit the training distribution. The reader's conditional verdict already points at this, but the paper has the control experiment available (w/o Distillation) and simply does not run it in the zero-shot regime, which is where 'general knowledge' should be visible. I would therefore keep the verdict conditional, with the explicit condition that the zero-shot no-distillation control be reported before the generalization claim is accepted. I do not call for rejection: the benchmark numbers may be valid and the method may work; the concern is that the causal attribution to VFM knowledge is currently under-evidenced. The gating-gradient observation in my attack is a secondary supporting point about internal consistency, not the primary basis for the verdict.","tokens_in":13401,"tokens_out":12461,"duration_ms":136308,"concrete_test":"Retrain the 'w/o Distillation' variant from Table 1 under the exact Table 3 protocol: pretrain on Scene Flow with the same 200k steps, batch size 8, crop 320×720, and one-cycle schedule, keeping all expert/alignment/selection modules and the forward-fusion path but omitting LKD (Eq. 6) from LAIO. Evaluate directly on Middlebury at the F and H resolutions used in Table 3 and report EPE and D1. If the gap between full and w/o-Distillation disappears in this zero-shot setting, the VFM-distillation explanation for the Table 3 gains is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central generalization claim—that VFM knowledge improves stereo matching beyond benchmark fitting—rests on the assumption that the distillation loss (Eq. 4) and gated expert fusion (Eqs. 7–8) transfer generic representations. Table 1 provides a capacity-matched control: 'w/o Distillation' appears to retain the expert networks, forward fusion, and selection but drops LKD, and full beats it on the MiddEval v3 training set (0.66/3.48 vs 0.72/3.87). However, the zero-shot result in Table 3 (Scene Flow → Middlebury) is never subjected to this control. If the w/o-Distillation variant matches full under the zero-shot protocol, then the training-set gains from distillation reflect fitting or regularization, not transferable VFM knowledge, and the claimed zero-shot generalization is unsupported. A related weakness is that Eq. (6) sums distillation losses over all three VFMs without weighting by the gating logits, so the 'selective' module gates only forward fusion (Eq. 8), not distillation gradients; the mechanism cannot literally prevent conflicting VFM knowledge from entering the backbone during training. The missing zero-shot distillation control is the most load-bearing gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AIO-Stereo, a stereo matching method that transfers knowledge from three vision foundation models (DINOv2, SAM, Depth Anything v2) into a CNN-based iterative stereo network (built on Selective-IGEV). The method introduces expert networks and feature alignment networks for heterogeneous feature adaptation, a dual-level knowledge utilization that combines feature distillation with a residual forward-fusion connection, and a gating mechanism with KeepTopK selection to fuse expert features per pixel. The authors evaluate on Scene Flow, Middlebury, ETH3D, and KITTI-2015, reporting state-of-the-art or comparable results, including first place on the Middlebury leaderboard and best published results on ETH3D, plus a zero-shot Scene Flow-to-Middlebury experiment.","tokens_in":13710,"tokens_out":3471,"duration_ms":39434,"significance":"If the reported gains are robust, the paper makes a useful empirical contribution by showing that carefully aligned and selectively fused VFM features can improve a modern iterative stereo matcher, particularly in dark and low-texture regions. The design is novel in combining multi-teacher distillation with gated forward fusion for stereo matching, and the zero-shot evaluation is a positive feature. The strengths include external benchmark comparisons, a clear ablation intent, and qualitative visualizations of per-pixel expert selection. However, the central claims are currently not fully supported: the ablations are run on a single training set without statistical significance, the forward-fusion ablation is confounded by also dropping one teacher, the distillation loss is not gated despite the 'selective' claim, and the leaderboard-level SOTA statements go beyond the numbers actually shown. These issues affect the paper's main generalization and state-of-the-art claims, so the contribution cannot be accepted as-is.","major_comments":[{"comment":"The 'w/o Forward Fusion' ablation is confounded: this row removes the forward fusion path but also removes Depth Anything v2 from the model (the Depth Anything v2 column is unchecked). Consequently, the reported drop from 0.66/3.48 to 0.67/3.52 cannot be attributed solely to removing forward fusion; it may reflect the loss of a teacher. The authors should run a controlled ablation that removes forward fusion while keeping all three VFMs, and also remove Depth Anything v2 while keeping forward fusion, to isolate the two effects. As reported, the dual-level knowledge utilization claim is not cleanly supported.","section":"Table 1"},{"comment":"The central generalization claim—that VFM distillation improves zero-shot performance—is never tested with the appropriate control. The ablation study in Table 1 is performed on the MiddEval v3 training set, where the model is fine-tuned on the target data, so improvements there could reflect fitting or regularization rather than transferable VFM knowledge. The zero-shot Scene Flow-to-Middlebury experiment in Table 3 only reports the full model. The authors should rerun the zero-shot protocol with the 'w/o Distillation' and 'w/o Selection' variants; if those variants match the full model under zero-shot transfer, the distillation gains observed on the training set do not support the claimed generalization benefit.","section":"Zero-Shot Generalization (Table 3)"},{"comment":"The 'selective' mechanism does not actually gate the distillation gradients. Eq. (6) sums the MSE distillation losses over DINO, SAM, and Depth Anything without any weighting by the gating logits computed in Eq. (7); the gating weights are applied only to the forward fusion path in Eq. (8). Therefore, conflicting or low-quality VFM knowledge can still propagate into the backbone during training through the distillation loss, undercutting the paper's stated motivation of preventing knowledge conflicts among VFMs. Either the distillation loss should be weighted by the gating probabilities, or the authors should explicitly justify why unweighted distillation does not reintroduce the conflicts the gating is designed to avoid.","section":"Eq. (6) and Eq. (8)"},{"comment":"The claims of ranking 1st on the Middlebury dataset and outperforming all published work on ETH3D are not substantiated by Table 2, which compares only a small hand-picked set of prior methods. No Middlebury or ETH3D leaderboard snapshot, date, or URL is provided, and 'all published work' is not a verifiable statement from the reported experiments. The authors should either provide the leaderboard evidence from the respective benchmark websites or soften the claims to 'state-of-the-art among the compared methods.'","section":"Abstract and 'Comparisons with State-of-the-art'"}],"minor_comments":[{"comment":"There is a typo: 'start-of-the-art' should be 'state-of-the-art.'","section":"Abstract"},{"comment":"The text contains typographical errors such as 'finstune' instead of 'fine-tune' and 'Selevtive-IGEV' instead of 'Selective-IGEV.'","section":"Comparisons with State-of-the-art"},{"comment":"The hyperparameter k in KeepTopK is never specified in the implementation details, and no sensitivity analysis is provided for it; the authors should report the value used and, ideally, an ablation.","section":"Eq. (7)"},{"comment":"The description of the feature alignment network as 'heavier' and initialized with a 'higher learning rate' is not quantified; the architecture sizes, learning rate multipliers, and decay factors should be listed.","section":"Implementation Details"},{"comment":"The distillation loss uses MSE between aligned expert features and VFM features, but the authors do not discuss whether the VFM features are L2-normalized or otherwise scaled; this can strongly affect MSE-based distillation and should be clarified.","section":"Eq. (4)"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is timely and the experimental setup follows the field's conventions, but the missing ablation controls and unsupported leaderboard claims are substantive. The zero-shot distillation control in particular is the kind of experiment the authors should be able to run without changing the method, so I see this as fixable within a revision rather than a rejection. I would also gently note that claims such as 'ranks 1st on Middlebury' are easily verified by reviewers and should be backed by a concrete leaderboard link or version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a serious attempt to inject VFM knowledge into stereo matching. The dual-level selective transfer module (distillation plus forward fusion, with per-pixel gating over three VFM experts) is new for this task, and the results are strong on paper: better than Selective-IGEV on Middlebury, ETH3D, and KITTI, with a clear zero-shot jump on Scene Flow to Middlebury. The ablations mostly support the components: adding VFMs helps, and removing selection hurts. The idea of using multiple complementary VFMs with a learned per-pixel gate is sensible, and the expert/alignment network design is a reasonable way to handle heterogeneous architectures.\n\nThe soft spots are real, and the stress-test concern holds up. The zero-shot table (Table 3) never includes the w/o-Distillation control. Since the distillation loss is the only part of the method that literally carries VFM knowledge into the backbone, the zero-shot gain could be coming from extra capacity or from task training rather than from distillation. Without that control, the paper's central generalization claim is not actually tested. Relatedly, Eq. (6) sums the three distillation losses without any gating weights, so the \"selective\" mechanism gates only forward fusion, not the distillation gradients. The paper talks about preventing conflicting VFM knowledge, but the mechanism as written does not do that during training. Also, Table 1's w/o Forward Fusion row drops Depth Anything at the same time, so it is confounded; the text's interpretation is not justified. Finally, \"ranks 1st on Middlebury\" and \"outperforms all published work on ETH3D\" are stronger than what Table 2 supports, since it compares to only a handful of methods.\n\nNone of these are fatal. The method is plausible, and the training-set ablations suggest distillation does something. But the missing zero-shot control is load-bearing, and the confounding row needs fixing. If a zero-shot w/o-Distillation variant matches the full model, the narrative about VFM generalization collapses, even if the training-set gains remain.\n\nThis paper is for stereo matching researchers, especially those exploring pretrained model transfer. It deserves a serious referee, but I would ask the authors for the zero-shot distillation ablation and a deconfounded ablation table before trusting the headline claims.","headline":"Plausible method with real benchmark gains, but the zero-shot improvement is never isolated from the distillation loss that supposedly causes it.","tokens_in":14176,"tokens_out":3632,"would_cite":false,"duration_ms":40073,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AIO-Stereo claims that transferring selected knowledge from multiple vision foundation models into a CNN stereo network yields state-of-the-art disparity estimation, including first place on Middlebury and best published scores on ETH3D.","keywords":["stereo matching","vision foundation models","knowledge distillation","dual-level selective knowledge transfer","mixture of experts","feature alignment","zero-shot generalization","disparity estimation"],"falsifier":"A direct test would be to replace the learned gating with fixed uniform averaging of the three aligned expert features while keeping all other components identical. If the Middlebury and ETH3D gains persist, the selective mechanism is not responsible; if they collapse, the per-pixel selection carries the result. Alternatively, if zero-shot accuracy on a clearly new domain drops to the baseline, the transferred features are not general.","tokens_in":13251,"feed_emoji":"🧩","tokens_out":7236,"duration_ms":66765,"temperature":0.7,"pith_summary":"AIO-Stereo aims to show that stereo matching improves when the feature encoder, not just the iterative refinement, is strengthened by knowledge from vision foundation models (VFMs). The paper proposes a framework that distills DINOv2, SAM, and Depth Anything v2 into a single CNN-based stereo network, aligning the heterogeneous feature spaces and letting a gating network choose the most useful expert features per pixel. The authors report that this selective transfer puts the method first on the Middlebury benchmark and ahead of all published methods on ETH3D, and that it improves zero-shot performance on unseen real data. If the claim holds, strong stereo accuracy can come from better encoded features rather than from redesigning the update module.","feed_headline":"AIO-Stereo ranks first on Middlebury by distilling three vision models","feed_subtitle":"Fusing DINO, SAM, and Depth Anything features sharpens stereo disparity maps and tops the Middlebury leaderboard.","key_machinery":"The load-bearing component is the dual-level selective knowledge transfer (DLSKT) module. It operates at two levels: a distillation level, where an expert network plus a feature-alignment network maps each VFM's stage features into the CNN's latent space under an MSE loss $L_{\\mathrm{KD},i}$; and a fusion level, where the gating network $g_i = \\mathrm{KeepTopK}(\\mathrm{Softmax}(G_i(f_i)), k)$ computes per-pixel weights and the selected expert features are added into the residual stream, $f_{i+1} = B_i(f_i) + \\sum_{x \\in \\{d,s,a\\}} e^x_i \\odot g_i(x)$. This design is what lets heterogeneous Transformer-based VFM features be transferred into the CNN backbone while avoiding conflicts among conflicting knowledge sources.","core_discovery":"The paper's central claim is that the quality of the encoded features matters as much as the iterative optimization in stereo matching, and that a single model can absorb complementary strengths of multiple vision foundation models. AIO-Stereo builds on the Selective-IGEV baseline and inserts a dual-level selective knowledge transfer module into the context network's residual blocks. Each of three VFMs (DINOv2 for robust foreground and semantic features, SAM for edges and small objects, Depth Anything v2 for dark and low-texture areas) has its own expert and feature-alignment network; one MSE distillation loss aligns the heterogeneous feature spaces, and a gating network with per-pixel top-k selection fuses only the most relevant experts. The resulting model ranks first on Middlebury, achieves the best published results on ETH3D on most metrics, and improves D1-fg on KITTI-2015, all without changing the iterative refinement machinery.","pith_inferences":["The same selective multi-teacher transfer could be applied to other dense prediction tasks such as optical flow, monocular depth estimation, or semantic segmentation, where CNN encoders face similar feature-quality limits.","The per-pixel gating visualizations suggest a testable prediction: regions with dark or low texture should consistently route to Depth Anything, edges to SAM, and salient foregrounds to DINO; a quantitative region-wise study could confirm this division of labor.","As more specialized foundation models are released, the framework's benefit may grow, but the cost of running three VFM teachers during training, and the extra expert and alignment parameters, may become a bottleneck for real-time use.","Since the ablation validates the mechanism on a single training set, an independent multi-seed evaluation across several benchmarks would clarify how much of the gain is due to selection versus the added parameters."],"forward_implications":["Stereo matching can reach state-of-the-art accuracy by improving the encoder through VFM distillation, without redesigning the refinement module.","A single CNN backbone can simultaneously use several heterogeneous foundation models by aligning their features and per-pixel gating, resolving feature conflicts.","Zero-shot performance on unseen real datasets improves, indicating that the transferred features generalize beyond the training distribution.","The framework is flexible: adding or removing a VFM changes performance gracefully, so future foundation models can be plugged in without architectural changes."],"supporting_citations":[{"why":"Supplies the Selective-IGEV baseline and the comparison protocol; AIO-Stereo is built on it and reports relative improvements over it.","marker":"Wang et al. 2024"},{"why":"Supplies the RAFT-Stereo iterative architecture, correlation volumes, GRU update, and prediction loss that AIO-Stereo inherits.","marker":"Lipson, Teed, and Deng 2021"},{"why":"Provides DINO and DINOv2 self-supervised VFM features, one of the three teachers distilled into the model.","marker":"Caron et al. 2021; Oquab et al. 2023"},{"why":"Provides SAM features for edges and small objects, the second teacher.","marker":"Kirillov et al. 2023"},{"why":"Provides Depth Anything V2 features for depth cues in dark and low-texture regions, the third teacher.","marker":"Yang et al. 2024a,b"},{"why":"Motivates feature alignment for heterogeneous architectures, used to justify the alignment networks.","marker":"Hao et al. 2024"},{"why":"Supplies the mixture-of-experts gating idea behind the selective knowledge transfer.","marker":"Jacobs et al. 1991"},{"why":"Establishes the knowledge-distillation objective that the MSE distillation loss extends.","marker":"Hinton, Vinyals, and Dean 2015"}],"fun_headline_variants":["AIO-Stereo fuses three vision models to top Middlebury","One model, three foundations: AIO-Stereo leads stereo matching","Selective multi-VFM transfer pushes AIO-Stereo to leaderboards"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the learned feature alignments and per-pixel gating genuinely transfer general visual knowledge from the three foundation models, rather than only fitting the specific training and validation benchmarks that produced the reported leaderboard numbers.","fun_headline_variants_meta":{"raw":{"variants":["AIO-Stereo fuses three vision models to top Middlebury","One model, three foundations: AIO-Stereo leads stereo matching","Selective multi-VFM transfer pushes AIO-Stereo to leaderboards"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000607,"raw_usage":{"total_tokens":2809,"prompt_tokens":904,"completion_tokens":1905,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":1850}},"tokens_in":520,"tokens_out":1905,"duration_ms":15195,"temperature":1.0,"reasoning_tokens":1850,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:34:22.713997+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would be to replace the learned gating with fixed uniform averaging of the three aligned expert features while keeping all other components identical. If the Middlebury and ETH3D gains persist, the selective mechanism is not responsible; if they collapse, the per-pixel selection carries the result. Alternatively, if zero-shot accuracy on a clearly new domain drops to the baseline, the transferred features are not general.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the RAFT-Stereo iterative architecture, correlation volumes, GRU update, and prediction loss that AIO-Stereo inherits."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates feature alignment for heterogeneous architectures, used to justify the alignment networks."}],"review_version":1}