{"id":"a16827ac-002e-4d64-8042-6dd1a941ec6e","arxiv_id":"2412.06439","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":10,"one_line_summary":"A local attention transformer used as the final upsampler reduces optical flow endpoint error on Sintel Clean for RAFT, GMA, and FlowFormer.","lead":"The paper improves the final upsampling step of optical flow models by replacing convex upsampling with a local attention transformer, reporting lower endpoint error on Sintel and FlyingThings3D. The change is designed as a drop-in module for most current optical flow architectures, including RAFT, GMA, and FlowFormer.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline GMA gain (1.31→1.18) in the abstract comes from GMA+ALL-aug, yet no -AUG control exists for the original upsampler; without it, the claimed reduction cannot be attributed 'solely' to the convex upsampler.","rationale":"The reader's weakest assumption is that the 'recomputed' baselines may not have been trained under the same protocol as the TCU models. That is a valid concern, but the more direct problem is internal to the reported numbers: the abstract's GMA improvement uses GMA+ALL-aug, and no baseline with the original upsampler plus -AUG exists. Even if the baselines were perfectly recomputed, the 1.18 number would still conflate the TCU architecture with the -AUG training scheme. This is not a fundamental flaw in the method; the component-wise improvements in Table 2 and the consistency across RAFT, GMA, and FlowFormer give the approach plausibility. The concern is about attribution and missing controls, not correctness of the architecture. A single missing-control experiment would settle whether the headline GMA claim is valid. The paper also lacks released code and error bars, but those are addressable rather than fatal. Since the reader already issued a conditional verdict, this analysis does not change the verdict; it sharpens the specific condition that should be imposed: provide the -AUG baseline with the original upsampler, or revise the claim to exclude -AUG from the 'solely the upsampler' statement.","tokens_in":13083,"tokens_out":6000,"duration_ms":59635,"concrete_test":"Run the missing control: take released GMA with the original convex upsampler, fine-tune it for 100K iterations under the paper's stated protocol (pre-trained weights, LR 1e-4, batch size 3, C+T data), then apply the -AUG phase for 40K iterations with interpolation-based augmentations disabled, and evaluate Sintel Clean. Also evaluate the same baseline without the -AUG phase. If the original-upsampler GMA with -AUG reaches about 1.18, the headline gain is due to the training scheme rather than TCU; if it stays near 1.21 or 1.31, TCU retains credit. A similar control for RAFT and FlowFormer would confirm which portion of each reported gain is upsampler-specific.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim's GMA number is not attributable to the upsampler. Table 3 shows GMA+ALL (TCU without -AUG) at 1.21 and GMA+ALL-aug at 1.18; the abstract cites 1.18. Section 4 defines -AUG as an additional 40K fine-tuning phase with interpolation-based augmentations disabled—a training-scheme change, not a change to the upsampler. There is no row for GMA with the original convex upsampler plus -AUG, so the 0.13 reduction from the 1.31 baseline cannot be decomposed. The same table shows -AUG alone moves TCU-GMA by 0.03, so the training scheme is not negligible. Moreover, the abstract's three headline numbers come from three different configurations (RAFT+ALL, GMA+ALL-aug, FlowFormer+ALL), undercutting the 'solely adapting the convex upsampler' framing. A second, compounding gap is that 'recomputed' baselines are not described as following the same 100K fine-tune protocol; if they are merely released checkpoints, extra fine-tuning alone could explain part of the gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Transformers for Convex Upsampling (TCU), a drop-in replacement for the convex upsampler used in RAFT-style optical flow networks. The method reformulates convex upsampling as local (neighborhood) attention, which decouples mask size from parameter count and enables hierarchical 2x upsampling steps with multi-scale image features. The authors also propose decoupling the final upsampler's weights and an additional training phase (-AUG) that disables interpolation-based augmentations. On the C+T training setting, they report reduced Sintel Clean training-split EPE for RAFT (1.42 to 1.26), GMA (1.31 to 1.18), and FlowFormer (0.94 to 0.90), attributing the gains to adapting the convex upsampler.","tokens_in":13374,"tokens_out":6552,"duration_ms":61515,"significance":"The core reformulation of convex upsampling as neighborhood attention is elegant and technically plausible; it cleanly decouples mask size from parameter count and makes hierarchical upsampling natural. The paper addresses an under-explored component of optical flow pipelines and provides experiments across three architectures, with a useful honesty about the risk that the changes can hurt cross-dataset generalization (Section 6). However, the central attribution claim is weakened by confounds: the best GMA number includes a training-scheme change, the FlowFormer comparison uses a reduced configuration, and the baseline protocol is not specified. With additional control experiments or a more cautious claim, the contribution would be valuable.","major_comments":[{"comment":"The abstract's headline for GMA (1.31 to 1.18) is the GMA+ALL-aug row of Table 3, but that row includes the -AUG training scheme defined in Section 4 (an additional 40K fine-tuning phase with interpolation-based augmentations disabled), not just a change of the upsampler. No comparison is provided for GMA with the original convex upsampler under the -AUG protocol, and Table 2 shows -AUG alone changes Sintel Clean EPE by 0.03 for the TCU model, so the 0.13 improvement cannot be attributed solely to the upsampler. Moreover, the three headline numbers come from different configurations: RAFT+ALL, GMA+ALL-aug, and FlowFormer+ALL, so the claim 'by solely adapting the convex upsampler' is not supported by the reported experiments.","section":"Abstract; Section 4; Table 3"},{"comment":"It is unclear whether the 'recomputed' baselines in Table 3 (RAFT, GMA, FlowFormer) were re-trained under the same fine-tuning protocol described in Section 5 (100K iterations, learning rates 1e-4/2e-4, batch size 3) or are simply released checkpoints re-evaluated. If the latter, the improvement of the proposed models over these baselines could be partly an effect of additional fine-tuning. The paper should state explicitly how each baseline row was produced; if they are re-trained, give the exact protocol, and if they are not, a control fine-tuning of the original upsampler under the same protocol is needed.","section":"Section 5; Table 3"},{"comment":"The FlowFormer+ALL result is reported with the caveat that the transformer models are not fine-tuned and only scales 1/4 and 1/8 are used because scale 1/2 cannot be obtained. This means the comparison is not a clean replacement of the upsampler while keeping everything else fixed. Since the abstract cites the FlowFormer improvement as evidence for the upsampler's effect, the experiment should either be run with the same feature scales and fine-tuning protocol as the baseline, or the claim should be restricted to the configurations that are actually matched.","section":"Table 3 (FlowFormer rows and note)"}],"minor_comments":[{"comment":"The notation for mask dimensions is inconsistent: Eq. (12) states masks are in R^{f^2 m^2}, while Eq. (13) writes R^{f^2*m*m} and then equates it to R^{f^2 m^2}; please use a single consistent representation throughout.","section":"Section 3.1, Eqs. (12)-(13)"},{"comment":"The statement that 'all our proposed changes result in improvements' is contradicted by Table 2, where +DC+FT (1.24 Sintel Clean, 9.53 FlyingThings3D) is worse than +DC alone (1.23 and 9.38); please qualify this claim.","section":"Section 5.2, paragraph 2"},{"comment":"The claim that 'all public top submissions to the Sintel leaderboard show bilinear interpolation artifacts' is made without a citation or a supporting figure; please add evidence or soften the claim.","section":"Section 4"},{"comment":"The note about not fine-tuning the transformer models is important enough to be discussed in the main text, not only as a table footnote, because it directly affects the interpretation of the FlowFormer comparison.","section":"Table 3, FlowFormer note"},{"comment":"Reference [7] for Neighborhood Attention is missing the publication venue; please provide the full citation.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is promising, but the abstract's 'solely adapting the convex upsampler' claim currently exceeds the evidence. A GMA baseline with the original upsampler under the -AUG scheme, plus explicit specification of the baseline training protocol, would substantially strengthen the paper. The FlowFormer caveat also needs to be addressed before the headline claim can be taken at face value."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi [Name], quick take on arXiv:2412.06439. The central observation is that convex upsampling is local softmax attention; that's a genuinely useful reframing and it lets the authors scale the mask size without adding parameters. The hierarchical 2x upsampling and decoupled final weights are reasonable engineering. The results are consistent—RAFT, GMA, FlowFormer all improve on Sintel Clean train with the TCU upsampler—and the detail-bucket analysis is a nice diagnostic.\n\nBut the abstract's claim 'by solely adapting the convex upsampler' does not hold as written. The GMA number it cites, 1.18, is GMA+ALL-aug, which includes the -AUG training scheme (an extra 40K iterations with interpolation augmentations off). That's a training change, not an upsampler change. Table 3 shows GMA+ALL (TCU without -AUG) at 1.21, so -AUG moves the needle by 0.03 on the TCU model. There is no GMA+original-upsampler+-AUG row, so you can't decompose the 0.13 gap between 1.31 and 1.18. The other two headline numbers come from different configurations (RAFT+ALL, FlowFormer+ALL). So 'solely' is an overstatement.\n\nA second, compounding issue is the 'recomputed' baselines. The paper says all sessions fine-tune for 100K with specified learning rates, but doesn't say whether the RAFT/GMA/FlowFormer baselines were recomputed under that exact protocol or are just released checkpoints. If they're released checkpoints, then part of the gap could be from extra fine-tuning, not the upsampler. That's a load-bearing gap for the headline.\n\nAlso, there are no error bars, which is common but worth mentioning; and the numbers are on Sintel train, not test, which is a fitting measure, not generalization. The paper is actually honest about the Sintel Final degradation and cross-dataset trade-offs, which I appreciate.\n\nThe core idea is still interesting and the paper is worth engaging. The authors should be asked to (1) add the missing control—original upsampler with -AUG, and ideally the same fine-tuning protocol for all baselines—and (2) tone down the 'solely' language. If those are addressed, the claim that a larger, attention-based upsampler helps is likely to hold.\n\nWho's it for? Anyone working on optical flow architectures or upsampling modules, and people interested in attention-conv bridges. I'd send it to a serious referee, but the revision needs to fix the attribution before acceptance.","headline":"The convex-upsampling-as-attention idea is clean and the gains are likely real, but the abstract's 'solely adapting the convex upsampler' overstates the evidence because the GMA number includes a training-scheme change with no control.","tokens_in":13925,"tokens_out":3352,"would_cite":true,"duration_ms":31297,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Replacing the standard convex upsampler with a local-attention transformer improves optical flow accuracy across RAFT, GMA, and FlowFormer, with only the final upsampling step changed.","keywords":["optical flow","convex upsampling","neighborhood attention","local attention","transformers","end-point error","hierarchical upsampling","data augmentation"],"falsifier":"Run a controlled comparison where the only variable is the upsampler: start all models from identical pre-trained weights, apply the exact same fine-tuning protocol with the same number of iterations and learning rates, and vary only the final upsampling module; if the end-point-error gap disappears, the reported gains come from the training setup rather than from attention-based upsampling.","tokens_in":12864,"feed_emoji":"🖼️","tokens_out":5997,"duration_ms":51796,"temperature":0.7,"pith_summary":"The paper targets a small but widely shared component of modern optical flow networks: the convex upsampler that turns a low-resolution flow estimate into a full-resolution one. It argues that the standard 3x3 convex mask is unnecessarily restrictive, because a correct full-resolution pixel value exists only if the low-resolution neighborhood can be combined to produce it. The central move is to reformulate convex upsampling as local neighborhood attention, which decouples mask size from the number of parameters and allows hierarchical upsampling in three 2x steps. With only the upsampler replaced, plus a final training stage that disables interpolation-based augmentation, the authors report Sintel Clean training end-point-error dropping from 1.42 to 1.26 for RAFT, 1.31 to 1.18 for GMA, and 0.94 to 0.90 for FlowFormer. A careful reader would care because the change is presented as a drop-in replacement applicable to nearly every current state-of-the-art flow architecture.","feed_headline":"Local-attention upsampler cuts optical flow error across models","feed_subtitle":"Swapping only the last upsampling step trims Sintel Clean EPE for RAFT, GMA, and FlowFormer.","key_machinery":"The central object is the Transformers for Convex Upsampling (TCU) module, a drop-in replacement for the convex upsampler. It is built on the equivalence between convex upsampling and Neighborhood Attention: the softmax-weighted dot product with low-resolution neighbors is identical in both, so local attention maps can serve as convex masks while their size is decoupled from the number of parameters. The module performs three 2x upsampling steps with mask sizes 9, 7, and 5, concatenates context and image features at each scale, and is applied only to the final refinement iteration's flow, giving that last step its own weights. This machinery carries the argument because each proposed change targets a specific failure mode: larger masks make a correct convex combination more likely to exist, hierarchical upsampling reduces the difficulty of aligning sub-pixels, extra features align flow with edges, and decoupling avoids polluting the final upsampler with noisy intermediate flow estimates.","core_discovery":"The paper's central claim is that convex upsampling, widely used to raise optical flow from one-eighth to full resolution, is best understood and improved as local attention. The authors observe that the softmax-weighted dot product over a sliding window of low-resolution flow, which defines a convex combination, is the same operation as neighborhood attention: both take a dot product between a normalized weight map and local values. They introduce Transformers for Convex Upsampling (TCU), which uses local attention maps as convex masks, allowing larger mask sizes, three hierarchical 2x upsampling steps instead of one 8x step, and the injection of image features at intermediate resolutions. They also decouple the upsampler used for the final refinement iteration from the shared upsampler used for intermediate iterations, and propose a final training stage with interpolation-based augmentations disabled to remove bilinear interpolation artifacts. On the FlyingChairs plus FlyingThings3D training setting, they report reducing the Sintel Clean training end-point-error of RAFT from 1.42 to 1.26, GMA from 1.31 to 1.18, and FlowFormer from 0.94 to 0.90, solely by adapting the upsampler.","pith_inferences":["The same convex-upsampling-as-attention argument applies to other dense prediction tasks that upsample low-resolution outputs, such as depth estimation, semantic segmentation, or scene flow, where a TCU-style module could improve edge fidelity.","The paper's 'existence of a convex combination' framing suggests a quantitative diagnostic: one could measure per patch whether the ground-truth high-resolution flow lies in the convex hull of the low-resolution neighborhood, and use that to decide where larger masks help most.","The authors' caution about cross-dataset generalization implies a testable prediction: on blurry datasets like Sintel Final, sharp edge alignment from larger masks will hurt unless the training data also contains blur, so the optimal mask size may be dataset-dependent.","The reported gains assume the baselines were fine-tuned under the same protocol; a direct re-run with identical pre-training, learning rate, and iteration counts would settle whether the upsampler alone, rather than training details, is responsible for the improvement."],"forward_implications":["Any RAFT-style optical flow model that currently uses the original 8x convex upsampler can adopt TCU in its final refinement step without retraining the rest of the network, so the reported gains are portable.","Larger convex masks are not merely a parameter-count issue: they increase the chance that a correct high-resolution flow value is representable as a convex combination of low-resolution neighbors, so further gains from even larger masks should appear where memory permits.","Hierarchical 2x upsampling with multi-scale image features may become the default design for flow upsampling, replacing the single 8x step.","The -AUG training scheme shows that bilinear interpolation artifacts in augmented training data are baked into model outputs; disabling interpolation-based augmentation at the end sharpens edges and improves clean-dataset accuracy, though it can hurt generalization to blurry targets such as Sintel Final.","Because the upsampler is a small, largely independent part of the network, its improvements compose with other advances in flow prediction, which explains the consistent gains across RAFT, GMA, and FlowFormer."],"supporting_citations":[{"why":"Introduces convex upsampling and the baseline RAFT architecture that the proposed TCU replaces and evaluates against.","marker":"[30]"},{"why":"Provides Neighborhood Attention, the formulation used to rephrase convex upsampling and to build the local attention masks.","marker":"[7]"},{"why":"The GMA optical flow model used as a baseline and integration testbed.","marker":"[14]"},{"why":"The FlowFormer model used as a baseline and integration testbed.","marker":"[9]"},{"why":"Introduces 3-step 2x hierarchical upsampling within RAFT-style models, which the paper adapts for TCU.","marker":"[13]"},{"why":"The FlyingThings3D dataset used for training and for the high-detail error analysis.","marker":"[19]"},{"why":"The Sintel dataset whose Clean and Final splits are the main evaluation benchmarks.","marker":"[2]"},{"why":"The FlyingChairs dataset used in the C+T training protocol.","marker":"[4]"}],"fun_headline_variants":["Local attention upsampler lowers optical flow error","Rethinking convex upsampling as local attention cuts error","Attention upsampler swap trims flow error on three models","Transformer upsampler reduces flow error with local attention"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported improvements assume the comparison models were retrained under exactly the same fine-tuning protocol as the models using the new upsampler, so that the upsampler is the only meaningful difference.","fun_headline_variants_meta":{"raw":{"variants":["Local attention upsampler lowers optical flow error","Rethinking convex upsampling as local attention cuts error","Attention upsampler swap trims flow error on three models","Transformer upsampler reduces flow error with local attention"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00137,"raw_usage":{"total_tokens":5618,"prompt_tokens":1074,"completion_tokens":4544,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":690,"completion_tokens_details":{"reasoning_tokens":4479}},"tokens_in":690,"tokens_out":4544,"duration_ms":30661,"temperature":1.0,"reasoning_tokens":4479,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:39:12.211398+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled comparison where the only variable is the upsampler: start all models from identical pre-trained weights, apply the exact same fine-tuning protocol with the same number of iterations and learning rates, and vary only the final upsampling module; if the end-point-error gap disappears, the reported gains come from the training setup rather than from attention-based upsampling.","supporting_citations":[{"cited_title":"A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation","cited_arxiv_id":null,"evidence_quote":"The FlyingThings3D dataset used for training and for the high-detail error analysis."},{"cited_title":"Flownet: Learning optical flow with convolutional networks","cited_arxiv_id":null,"evidence_quote":"The FlyingChairs dataset used in the C+T training protocol."},{"cited_title":"Raft: Recurrent all-pairs field transforms for optical flow","cited_arxiv_id":null,"evidence_quote":"Introduces convex upsampling and the baseline RAFT architecture that the proposed TCU replaces and evaluates against."},{"cited_title":"Neighborhood attention transformer","cited_arxiv_id":null,"evidence_quote":"Provides Neighborhood Attention, the formulation used to rephrase convex upsampling and to build the local attention masks."},{"cited_title":"Learning to estimate hidden motions with global motion aggregation","cited_arxiv_id":null,"evidence_quote":"The GMA optical flow model used as a baseline and integration testbed."},{"cited_title":"FlowFormer: A Transformer Architecture for Optical Flow","cited_arxiv_id":"2203.16194","evidence_quote":"The FlowFormer model used as a baseline and integration testbed."},{"cited_title":"Multi-scale raft: Combining hierarchical concepts for learning-based optical flow estimation","cited_arxiv_id":null,"evidence_quote":"Introduces 3-step 2x hierarchical upsampling within RAFT-style models, which the paper adapts for TCU."}],"review_version":1}