{"id":"2de99d60-da83-4f52-87f4-bb215fe9a93b","arxiv_id":"2501.01023","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"HART, a linear-complexity stereo transformer using Hadamard product attention with the Dense Attention Kernel, reports SOTA EPE 0.42 on Scene Flow and 1st place on KITTI 2012 reflective areas at submission.","lead":"HART is a new stereo matching transformer that computes attention with a Hadamard product and a dense activation kernel, cutting cost from quadratic to linear. It reports state-of-the-art results on reflective regions of the KITTI 2012 benchmark and strong zero-shot generalization, with code released.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Rank(A)/m metric in Tab. 8 is ill-posed (comparing n×n vs c×n matrices) and the ablations do not show rank tracks EPE, so DAK's low-rank-bottleneck claim is unsubstantiated.","rationale":"Agree with the reader that the weakest assumption is the meaningfulness of Rank(A)/m as a proxy for capacity. I sharpen this: the metric is not even comparable across rows because the matrix shapes differ, and the paper's own ablations (rows 4 vs 5) show rank does not track EPE. The central claim of the paper, that DAK overcomes the low-rank bottleneck, is therefore not established. The empirical SOTA results are credible and the code is available, so the paper can stand as a strong baseline if the causal language is softened. The proposed SVD-truncation experiment is a direct falsification test of the bottleneck claim; the rank-redefinition check settles whether Tab. 8's evidence is an artifact of the metric.","tokens_in":16069,"tokens_out":8796,"duration_ms":78861,"concrete_test":"Use the released code to run the full HART (Tab. 8 row 6) on Scene Flow, but before applying DAK in Eq. 4, replace A with its rank-r truncated SVD for r ∈ {8, 16, 32, 64} (keeping everything else fixed). Measure EPE and the reported Rank(A)/m. If EPE does not degrade as r is reduced well below the natural full rank of the c×n matrix, the low-rank bottleneck is not the operative mechanism, and the paper's causal story fails. Additionally, recompute Tab. 8's Rank(A)/m with a common definition (rank of the c×n weight matrix divided by min(c,n) for all rows, including vanilla SA re-expressed in the same c×n gate form) to verify whether DAK still increases effective rank under a fair comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's causal claim—that DAK removes the upper bound on attention weights and thereby alleviates the low-rank bottleneck, improving matching in reflections (Sections 1, 3.2.2, Eq. 4)—rests on the Rank(A)/m metric in Tab. 8. This metric is not well-defined across the compared architectures. For the vanilla self-attention baseline (row 2), the attention matrix is n×n (m=n); for the Hadamard-product variants (rows 3–6), the 'attention matrix' A = ||Q||^2 ⊙ ||K||^2 is c×n (m=c). Comparing rank ratios of matrices with different shapes and semantics is not meaningful, and the known low-rank result for softmax attention (rank ≤ head dimension) does not transfer to this c×n elementwise product. Moreover, the ablations contradict the rank→accuracy link: row 4 (DAK, no MKOI) has Rank(A)/m=93.0 but EPE=0.46, worse than row 2 (vanilla SA, rank 78.1, EPE 0.44); row 5 (Softmax + MKOI, rank 81.8) achieves EPE 0.43, better than row 4 despite lower rank. Thus the reported improvements are driven by MKOI and the global architecture, not by DAK's rank effect. Mathematically, DAK(A)=1+ELU(A) (Eq. 5), so the 'dense' kernel is just an ELU activation with a skip connection; without a controlled experiment linking rank to EPE, the low-rank-bottleneck narrative is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"HART proposes a stereo matching transformer built on Hadamard-product self-attention (HPSA), with two main components: the Dense Attention Kernel (DAK), which replaces softmax with a positive-valued activation to supposedly overcome a low-rank bottleneck, and the Multi-Kernel & Order Interaction (MKOI) module, which adds spatial and channel interactions via channel splitting and multi-scale convolutions. The encoder is paired with an IGEV-style recurrent decoder. The paper reports SOTA or near-SOTA results on Scene Flow, KITTI 2012/2015, Middlebury, and zero-shot Driving Stereo, including first place on the KITTI 2012 reflective benchmark at the time of submission. The central causal claim is that DAK raises the rank of the attention matrix, reduces feature collinearity, and thereby improves matching in ill-posed regions such as reflections.","tokens_in":16333,"tokens_out":4423,"duration_ms":42894,"significance":"If the causal story were supported, HART would be a notable contribution: it offers linear-complexity attention for stereo transformers, strong empirical results across multiple benchmarks, zero-shot generalization experiments, and an extension replacing DLNR's attention module. The manuscript ships code and follows standard training protocols, which is a strength. However, the paper's load-bearing theoretical claim—that DAK alleviates a low-rank bottleneck and that this mechanism drives the accuracy gains—is not established by the evidence presented. The ablation metric used to support the claim is not well-defined across architectures, and the ablations do not show that rank tracks accuracy. The empirical contribution is real, but the causal narrative needs either rigorous support or substantial softening.","major_comments":[{"comment":"The Rank(A)/m metric is not a well-defined cross-architecture comparison. In Experiment 2, the attention matrix for vanilla SA is n×n, while in Experiments 3–6 the Hadamard-product attention matrix is c×n; comparing rank ratios of matrices with different shapes and different semantic roles is not meaningful. In addition, the known low-rank bound for softmax attention (rank at most the head dimension) does not transfer to the c×n elementwise product used here. To support the claim that DAK alleviates a low-rank bottleneck, the authors need to define a metric on the actual attention operator used in each variant and verify that higher rank is causally tied to lower EPE.","section":"§4.4, Table 8"},{"comment":"The ablation results contradict the rank-to-accuracy link. Row 4 (DAK without MKOI) has Rank(A)/m=93.0 but EPE=0.46, which is worse than row 5 (Softmax with MKOI) at Rank(A)/m=81.8 and EPE=0.43; row 5 achieves better accuracy despite a lower rank. Within the same MKOI setting, replacing Softmax with DAK changes EPE only from 0.43 to 0.42. This pattern indicates that MKOI and the overall architecture, rather than DAK's rank effect, are the main drivers of the reported gains. The paper should provide a controlled experiment that isolates DAK's effect on rank and matching accuracy, or explicitly drop the rank-based causal claim.","section":"§4.4, Table 8"},{"comment":"Equation (6) is not a valid derivation of a DAK-specific benefit. Since DAK(A)=1+ELU(A), Eq. (5) is simply a residual ELU, and Eq. (6) is the standard chain rule for a residual connection; the expression inside MKOI mixes a scalar with a derivative of a sum and is dimensionally unclear. The paper presents this as a theoretical justification for DAK, but it does not establish that DAK removes a rank bottleneck. Either provide a rigorous argument linking DAK to rank improvement or temper the theoretical framing.","section":"§3.2.2, Eq. (6)"},{"comment":"The parameter-count argument '2nc/d << n^2' does not prove a low-rank bottleneck. The rank of the attention matrix is bounded by its dimensions and the algebraic structure of Q and K, not by the number of parameters in the query and key matrices. The paper should state precisely which matrix's rank it analyzes, before or after DAK/Softmax, and what exact rank bound DAK is claimed to escape; otherwise the low-rank-bottleneck narrative is unsupported.","section":"§3.2.1 and §3.2.2"}],"minor_comments":[{"comment":"The heading 'Caculation of Attention Matrix' contains a typo; it should read 'Calculation of Attention Matrix'.","section":"§3.2, heading"},{"comment":"There are duplicated words in the text, e.g., 'and and secured a position' in the Introduction; the manuscript needs a careful proofread.","section":"Abstract and Introduction"},{"comment":"It is unclear whether the Rank(A)/m metric in Table 8 is computed on A before or after applying DAK/Softmax and MKOI; this should be stated explicitly, along with the number of inference steps and samples used to compute the average.","section":"§3.2.2 and §4.4"},{"comment":"The phrase 'the derivative properties of DAK determine its ability to further widen the attentional gap' is vague; please specify the derivative inequality and how it redirects attention.","section":"§3.2.2"},{"comment":"The notation in Eq. (7) is dense and the kernel-size indexing s=2m+3 is not defined in the text around the equation; adding a short explanation would improve readability.","section":"§3.3, Eq. (7)"},{"comment":"The claim 'HART ranked 1st at the time of submission' is time-dependent; the paper should note that leaderboard positions may change and the evaluation is as of the submission date.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The empirical work is solid and the code availability is a plus, but the central theoretical contribution—the low-rank-bottleneck alleviation by DAK—is currently unsupported by the evidence in the manuscript. The authors should be pushed to either provide a rigorous and correctly defined rank analysis with controlled ablations, or substantially reword the claims so that DAK is presented as an empirically motivated activation choice rather than a mechanism with a proven rank effect. This is fixable within the scope of the paper, but it is load-bearing and cannot be left as is."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: HART is a legitimate and useful stereo transformer baseline with strong empirical results and released code, but you should not send anyone to it for the theory. The causal claim that DAK fixes a low-rank bottleneck is the weakest part of the paper, and the evidence in the ablations actually cuts against it.\n\nWhat's actually new: the Hadamard-product attention with linear complexity, the DAK activation (1+ELU), and the MKOI multi-kernel interaction module. The combination is new for stereo matching, and the paper demonstrates consistent gains on Scene Flow, KITTI, Middlebury, and Driving Stereo, including first place on the KITTI 2012 reflective leaderboard at submission. The HART-DLNR extension is a nice touch; it shows the attention module transfers to another transformer and cuts inference time. The empirical work is extensive and the training protocol follows standard practice, so I'd trust the numbers more than most papers in this area. Code release is a real plus.\n\nWhere it's soft, and this matters more than the authors admit: the Rank(A)/m metric in Tab. 8 is not well-defined. For vanilla SA the attention matrix is n×n; for the Hadamard variants it's c×n. Comparing rank ratios of matrices with different shapes and semantics isn't meaningful. Worse, the ablations contradict the stated mechanism. Row 4 (DAK, no MKOI) has the second-highest rank (93.0) but the worst EPE (0.46) among transformer rows, worse than vanilla SA (78.1, 0.44). Row 5 (Softmax + MKOI) achieves 0.43 with a lower rank (81.8). So the only conclusion you can draw is that MKOI and the overall architecture drive the gains, not DAK's rank effect. The authors claim DAK reduces feature collinearity, but they don't measure collinearity directly and don't isolate the link to disparity accuracy. The gradient derivation around Eq. 6 is also pretty loose; it reads as a standard chain-rule identity dressed up to support DAK.\n\nOther soft spots are minor by comparison: no error bars or multiple seeds, a few typos, and the self-citation density is high but not disqualifying. The paper would benefit from a revised narrative that either directly tests the rank-accuracy relationship or drops the low-rank-bottleneck claim and presents DAK as a useful nonlinear activation.\n\nWho should read this: anyone working on efficient stereo transformers or looking for a strong open-source baseline. It deserves a serious referee; the empirical work is solid enough that the weak theory shouldn't sink it, but the authors should be asked to fix the metric and the causal claims before publication.","headline":"Solid engineering baseline with real benchmark gains, but the paper's central low-rank-bottleneck story is not supported by its own ablations and the Rank(A)/m metric is ill-posed.","tokens_in":16932,"tokens_out":1496,"would_cite":true,"duration_ms":15935,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that replacing softmax attention with an unbounded positive kernel built from a Hadamard product of query and key norms relieves the low-rank bottleneck in stereo transformers, yielding state-of-the-art disparity accuracy…","keywords":["Hadamard attention","stereo matching","linear transformer","low-rank bottleneck","dense attention kernel","multi-kernel interaction","recurrent stereo transformer","reflective surface matching"],"falsifier":"Train the same HART architecture with DAK replaced by another positive unbounded activation such as $\\mathrm{ELU}(x)+1$ or Softplus, keeping all other settings identical. If end-point error on Scene Flow and reflective-region errors stay essentially the same while $\\mathrm{Rank}(A)/m$ changes, then the low-rank-bottleneck story is not what carries the performance; if $\\mathrm{Rank}(A)/m$ rises but accuracy does not, the rank metric is not predictive.","tokens_in":15796,"feed_emoji":"🪞","tokens_out":6049,"duration_ms":50592,"temperature":0.7,"pith_summary":"HART is a stereo-matching transformer built on Hadamard-product attention that runs in linear time rather than quadratic time. The paper's central claim is that the Dense Attention Kernel (DAK), which maps attention weights to the unbounded interval $(0, +\\infty)$ instead of softmax's bounded range, relieves the low-rank bottleneck of self-attention and reduces feature collinearity, letting the model match points in ill-posed regions such as reflective surfaces. To compensate for the Hadamard product's lack of spatial and channel interaction, the Multi-Kernel & Order Interaction (MKOI) module mixes features across different kernel sizes and channel groups. The paper reports an end-point error of 0.42 pixels on Scene Flow, first place on the KITTI 2012 reflective leaderboard at the time of submission, and zero-shot generalization to Middlebury and Driving Stereo with lower errors than prior stereo transformers. If correct, the method offers a scalable attention design that could serve as a new baseline for stereo transformer models.","feed_headline":"A denser attention kernel tops reflective stereo matching","feed_subtitle":"Hadamard-product attention runs in linear time and ranks first on KITTI 2012 reflections.","key_machinery":"The load-bearing object is the Hadamard Product Self-Attention (HPSA) mechanism, whose attention matrix is computed as the element-wise (Hadamard) product of the squared norms of query and key, giving $O(n)$ complexity instead of $O(n^2)$. Dense Attention Kernel (DAK) is the activation that keeps every entry positive and unbounded, while Multi-Kernel & Order Interaction (MKOI) splits the channels into groups processed with kernels of sizes $1\\times 1$, $3\\times 3$, $5\\times 5$, and $7\\times 7$ to restore spatial and channel mixing that pure Hadamard products lack. Together these components replace softmax attention in the encoder, and the decoder follows an iterative update scheme with an LSTM-based refinement module inspired by recurrent stereo networks.","core_discovery":"The paper's core discovery is that replacing the matmul-plus-softmax attention in stereo transformers with an attention matrix $A = \\|Q\\|^2 \\odot \\|K\\|^2$, followed by a dense kernel $\\mathrm{DAK}(A) = A+1$ for $A\\ge 0$ and $e^{A}$ for $A<0$, produces a positive attention matrix that keeps all parameters active and improves disparity accuracy in difficult regions. The authors argue that removing the upper bound on attention weights lowers feature similarity and thereby counteracts the low-rank bottleneck identified for standard multi-head attention. The empirical anchor is an ablation study showing that the ratio $\\mathrm{Rank}(A)/m$ rises from 78.1% with vanilla self-attention to 94.5% with DAK plus MKOI, while end-point error on Scene Flow drops from 0.44 to 0.42 pixels and inference time falls from 0.71 seconds to 0.36 seconds.","pith_inferences":["Editorial inference: the paper's argument implies that any activation keeping attention weights positive and unbounded, such as $\\mathrm{ELU}(x)+1$ or Softplus, might capture part of the benefit; the unique contribution of DAK's piecewise form would need to be isolated by a controlled experiment the paper does not run.","Editorial inference: the $\\mathrm{Rank}(A)/m$ metric is a self-defined proxy; a stronger test would compute the numerical rank or effective rank with a tolerance and correlate rank increases with accuracy gains per region type rather than across a whole benchmark.","Editorial inference: because MKOI adds back local convolutions, HPSA is best understood as a convolution-attention hybrid, and the paper's claim of global interaction may hold only at the level of the full multi-scale encoder rather than within a single attention step."],"forward_implications":["Attention in stereo transformers can be made linear-time without sacrificing accuracy, removing a computational barrier to high-resolution and real-time stereo matching.","DAK's unbounded positive kernel could be dropped into other vision transformers that suffer from low-rank attention, potentially improving accuracy in ill-posed regions beyond stereo.","The rank-based diagnostic $\\mathrm{Rank}(A)/m$ gives practitioners a cheap way to compare attention mechanisms by representational richness, not just by downstream error.","Improved matching on reflective surfaces suggests safer disparity estimates for autonomous driving on glass, wet roadways, and other specular scenes."],"supporting_citations":[{"why":"Supplies the low-rank bottleneck concept in multi-head attention that DAK is designed to fix.","marker":"[9]"},{"why":"Defines the sequence-to-sequence transformer approach to stereo matching that HART builds on.","marker":"[7]"},{"why":"Provides the iterative decoder and correlation volume design (IGEV) that HART adapts for disparity refinement.","marker":"[5]"},{"why":"Establishes the recurrent all-pairs field transform paradigm used for iterative disparity updates.","marker":"[4]"},{"why":"A channel-attention stereo transformer baseline that HART compares against and whose attention module is later replaced by HPSA in the extension experiment.","marker":"[16]"},{"why":"Source of the gated feed-forward design and efficient attention ideas reused in HART's SGFF and HPSA components.","marker":"[18]"},{"why":"A recent recurrent stereo CNN whose motif-channel attention and LSTM-style updates inform HART's decoder design.","marker":"[6]"},{"why":"The Scene Flow dataset provides the training and test data for the paper's primary accuracy and ablation results.","marker":"[33]"}],"fun_headline_variants":["Hadamard attention tops reflective stereo matching","Unbounded attention weights improve stereo matching","Dense attention kernel improves stereo accuracy","Stereo transformer with denser attention ranks first","Hadamard-product attention speeds up stereo matching"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that the rank of the attention matrix, as measured by the paper's Rank(A)/m metric, is the real driver of matching accuracy rather than a side effect of DAK being a better activation function.","fun_headline_variants_meta":{"raw":{"variants":["Hadamard attention tops reflective stereo matching","Unbounded attention weights improve stereo matching","Dense attention kernel improves stereo accuracy","Stereo transformer with denser attention ranks first","Hadamard-product attention speeds up stereo matching"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000986,"raw_usage":{"total_tokens":4180,"prompt_tokens":943,"completion_tokens":3237,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":3170}},"tokens_in":559,"tokens_out":3237,"duration_ms":22930,"temperature":1.0,"reasoning_tokens":3170,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:36:45.476693+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same HART architecture with DAK replaced by another positive unbounded activation such as $\\mathrm{ELU}(x)+1$ or Softplus, keeping all other settings identical. If end-point error on Scene Flow and reflective-region errors stay essentially the same while $\\mathrm{Rank}(A)/m$ changes, then the low-rank-bottleneck story is not what carries the performance; if $\\mathrm{Rank}(A)/m$ rises but accuracy does not, the rank metric is not predictive.","supporting_citations":[{"cited_title":"Low-rank bottleneck in multi-head attention models","cited_arxiv_id":null,"evidence_quote":"Supplies the low-rank bottleneck concept in multi-head attention that DAK is designed to fix."},{"cited_title":"Revisiting stereo depth estimation from a sequence-to-sequence perspective with transformers","cited_arxiv_id":null,"evidence_quote":"Defines the sequence-to-sequence transformer approach to stereo matching that HART builds on."},{"cited_title":"Iterative geometry encoding volume for stereo matching","cited_arxiv_id":null,"evidence_quote":"Provides the iterative decoder and correlation volume design (IGEV) that HART adapts for disparity refinement."},{"cited_title":"Raft-stereo: Multilevel recurrent field transforms for stereo matching","cited_arxiv_id":null,"evidence_quote":"Establishes the recurrent all-pairs field transform paradigm used for iterative disparity updates."},{"cited_title":"High-frequency stereo matching network","cited_arxiv_id":null,"evidence_quote":"A channel-attention stereo transformer baseline that HART compares against and whose attention module is later replaced by HPSA in the extension experiment."},{"cited_title":"Restormer: E fficient transformer for high-resolution image restoration","cited_arxiv_id":null,"evidence_quote":"Source of the gated feed-forward design and efficient attention ideas reused in HART's SGFF and HPSA components."},{"cited_title":"Mocha-stereo: Motif channel attention network for stereo matching","cited_arxiv_id":null,"evidence_quote":"A recent recurrent stereo CNN whose motif-channel attention and LSTM-style updates inform HART's decoder design."},{"cited_title":"A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation","cited_arxiv_id":null,"evidence_quote":"The Scene Flow dataset provides the training and test data for the paper's primary accuracy and ablation results."}],"review_version":1}