{"id":"cd688eee-c9f8-4895-9eed-81c20db0c183","arxiv_id":"2411.14865","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Introduces KITTI-FC and GoPro-FC, the first corruption robustness benchmarks for optical flow, with 24 corruptions and 10 findings from 29 model variants.","lead":"This paper introduces the first benchmarks for testing how optical flow models cope with common image and video corruptions, adding seven temporal corruptions and two datasets. It evaluates 29 model variants and reports which models and corruption types are most robust, with implications for autonomous driving and video editing.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GoPro-FC conclusions rely entirely on RCRE, a no-ground-truth proxy for robustness that is validated only on KITTI-FC; if RCRE fails to rank models like CRE does in small-displacement settings, Observations 9-10 and Table 4 are unsupported.","rationale":"The reader's weakest-assumption analysis identified the static weather corruption rendering as the most load-bearing issue. That concern is legitimate: if Spatter, Fog, and Frost overlays are identical across a frame pair, correlation-based estimators may behave differently than they would under temporally varying weather, and this could bias weather-class robustness numbers and Observation 5. However, it affects only a subset of corruptions and leaves most of the benchmark intact. I find a more fundamental issue in the GoPro-FC evaluation methodology: all quantitative results for one of the two benchmarks rely on RCRE, a proxy whose equivalence to CRE-based robustness is demonstrated only on KITTI-FC, a large-displacement setting with ground truth. The paper's own evidence (Fig. 9d) is a model-level linear correspondence on KITTI-FC; it does not establish rank-order preservation in the small-displacement, no-ground-truth regime that defines GoPro-FC. Because Observations 9 and 10 are drawn from this benchmark, the risk is not merely a caveat but a possible reversal of conclusions if RCRE rankings diverge from true robustness rankings. The weather concern and the RCRE concern are both addressable empirically, and neither warrants rejection of the benchmark's release. The reader's conditional verdict remains appropriate, so the verdict should be unchanged. My recommendation is to prioritize the RCRE validation as a condition for accepting the GoPro-FC-specific claims, alongside the weather realism check and the training-step confound in the SAMFlow comparisons noted by the reader.","tokens_in":29388,"tokens_out":11905,"duration_ms":110172,"concrete_test":"Construct a small-displacement robustness set with ground truth, e.g., subsample FlyingThings or Sintel to pairs with maximum flow below 10 pixels, or synthesize pairs by warping images with small random flows. Apply the same 24 corruption types and severities, run the 11 supervised OOD models, and compute both CRE (Eq. 1, using ground truth) and RCRE (Eq. 3) for each model. Then compute the Spearman rank correlation between model-level RCRE and model-level CRE or corrupted-data EPE. If the rank correlation is below about 0.8, or if the best model changes, the GoPro-FC RCRE-based rankings (Table 4, Observations 9-10) are unreliable and should be reported with caveats or replaced by a pseudo-ground-truth metric such as distance to a strong reference flow.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative contribution of the GoPro-FC benchmark is its evaluation of robustness in small-displacement, video-editing scenarios without ground-truth flow. The only quantitative metric reported for GoPro-FC is RCRE (Eq. 3), defined as RCREc,s = ||fc,s - fclean||, the endpoint error between clean and corrupted predictions. This measures prediction shift, not error against ground truth. The paper justifies RCRE by reporting a linear relationship between RCRE and CRE on KITTI-FC (Fig. 9d), but that evidence comes from a large-displacement benchmark with ground truth and model-level aggregates; it does not establish that model rankings by RCRE match rankings by true EPE-based robustness in GoPro-FC's small-displacement regime. In small-displacement scenes, a model can produce a large prediction shift under corruption while staying near ground truth, or a small shift while crossing a motion boundary, and RCRE would misrank it. Since GoPro-FC has no ground truth, Table 4, Observation 9 (FlowFormer best), and Observation 10 (VFM does not help) rest entirely on this unvalidated proxy. The Limitations section (F.2) does not flag this, and no correlation statistic or rank-correlation for the small-displacement case is provided. This concern does not invalidate the released corrupted image pairs, but it does call into question the quantitative conclusions drawn from them.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces two corruption-robustness benchmarks for optical flow estimation, KITTI-FC and GoPro-FC, covering 24 corruptions in 6 classes, including 7 temporal corruptions, and proposes three robustness metrics: CRE, CREr, and RCRE. It evaluates 29 model variants from 15 optical flow methods under both Out-of-Domain and In-Domain settings for KITTI-FC and Out-of-Domain only for GoPro-FC, then derives 10 observations about model robustness, including the dependence of absolute robustness on clean estimation performance, the relative stability of unsupervised methods, the advantage of Transformer/SAM-based architectures in large-displacement settings, and the greater impact of corruptions that destroy local information. The benchmarks and code are promised for public release.","tokens_in":29718,"tokens_out":6463,"duration_ms":67359,"significance":"If the findings hold, this is a valuable and reusable evaluation suite for a relatively under-studied problem: corruption robustness of optical flow. The paper's strengths include the breadth of models evaluated (15 methods, 29 variants), the design of temporal corruptions specific to optical flow, the improved PSF blur simulation via designed low-quality lenses, the OOD/ID distinction, and the public release of the corrupted datasets. Several quantitative conclusions, however, are currently weaker than stated: the GoPro-FC rankings rest on a proxy metric validated only in a different displacement regime, the SAMFlow advantage is confounded with training budget, and Observation 1 is partly a consequence of the additive definition of CRE. The raw benchmark and tables are still a useful contribution, but the interpretive claims need tightening.","major_comments":[{"comment":"Observations 9 and 10, including the claim that FlowFormer is the best robust model in small-displacement settings, rest entirely on RCRE, a no-ground-truth proxy. The validation in Appendix D and Fig. 9(d) is performed on KITTI-FC, a large-displacement benchmark, and reports only model-level linear correspondence between CRE and RCRE. This does not establish that RCRE-based model rankings match ground-truth-based robustness rankings in the small-displacement GoPro-FC regime, where a model can shift its prediction substantially without moving far from the ground truth, or shift little while crossing a motion boundary. The Limitations section (F.2) does not flag this. I ask the authors to validate RCRE on a small-displacement subset with ground truth (for example, by subsampling KITTI or Sintel pairs), report per-corruption rank correlations between RCRE and CRE or EPE-based robustness, and clearly qualify the GoPro-FC quantitative conclusions as proxy-based if such validation is not possible.","section":"5.3, Table 4, Eq. (3), Appendix D"},{"comment":"The paper attributes SAMFlow's superior robustness to the vision foundation model (SAM), but SAMFlow differs from FlowFormer in training schedule as well as architecture: SAMFlow is trained for 240k steps with batch size 3, while FlowFormer is trained for 120k with batch size 6. A longer training budget alone could explain part of the observed improvement, and the paper states that SAMFlow's gradient accumulation makes batch 3 equivalent to batch 6, but the step-count difference remains. Please retrain FlowFormer/FlowFormer++ with matched training steps (or ablate SAM within the SAMFlow pipeline) before claiming that the VFM provides the robustness benefit, or alternatively soften Observation 2 to reflect the confound.","section":"5.2, Observation 2; Tables 5 and 6"},{"comment":"CRE is defined as EPE_c,s minus EPE_clean. Because EPE_clean appears as a subtractive term in the metric itself, a positive correlation between CRE and clean EPE is partly algebraic: even under a constant relative degradation, models with larger clean EPE will tend to have larger CRE. The paper's claim that 'absolute robustness depends heavily on estimation performance' therefore needs a decomposition that goes beyond the definitional component, for example by reporting partial correlations controlling for EPE_clean, or by showing that CREr (which removes this dependence) does not already account for the observed trend. As written, Observation 1 risks being tautological.","section":"4.1, Eq. (1); Observation 1"},{"comment":"For Spatter, Fog, and Frost, the same corrupted rendering is applied unchanged to both frames of the pair. Real weather artifacts are not static under camera and scene motion; an identical static overlay provides a zero-displacement matching clue that correlation-based optical flow estimators can exploit, potentially making weather robustness look better than it would be in practice and biasing model rankings. The paper's defense that these disturbances 'change slowly' is plausible for fog, but less so for spatter and frost, and it is not tested. I ask for a sensitivity analysis in which the overlay is independently sampled or slightly jittered between frames, with a report of whether the weather-class rankings and the overall observations change.","section":"3 and A.3, Weather corruptions"},{"comment":"Several headline conclusions are based on small numerical differences with no error bars or significance tests. For example, on GoPro-FC the average RCRE values of FlowFormer (1.29), FlowFormer++ (1.35), and CRAFT (1.42) differ by less than 0.15, and many KITTI-FC corruption columns show similarly close values. Since the observations claim ordinal rankings among specific models, the authors should report at least bootstrap confidence intervals or paired significance tests for the main aggregate comparisons, so that readers can judge whether the reported ordering is meaningful.","section":"Tables 2-4; Observations 9-10"}],"minor_comments":[{"comment":"The Introduction states that the authors found '7 interesting observations,' but the paper presents 10 observations. Please harmonize the count.","section":"1 vs. Abstract"},{"comment":"There are several typos: 'Hight-Light' should be 'High-Light' in Section 3; 'we threat it unchanged' should be 'we treat it unchanged' in Appendix A.3; and the Table 2 header 'A VGEPE' should be 'AVG EPE'.","section":"3 and A.3"},{"comment":"The claimed linear correspondence between CRE and RCRE is supported only by a scatter plot; please report the Pearson and Spearman correlation coefficients and, ideally, per-corruption values rather than only the model-level aggregate.","section":"Appendix D, Fig. 9(d)"},{"comment":"The column 'Average Rank' is not clearly defined: it is not obvious whether the rank is averaged across corruptions, across metrics, or across something else. Please specify the averaging procedure.","section":"Table 3"},{"comment":"The statement that CSFlow experiences 'an observed increase of up to 291% in CREr' is vague; please cite the specific corruption and condition that produce this maximum.","section":"Observation 5"},{"comment":"The Object Motion Blur simulation relies on FLAVR video interpolation, which assumes that the interpolated high-frame-rate frames are physically accurate enough for blur accumulation. This assumption is not validated and should be acknowledged as a limitation of that specific corruption.","section":"A.5, Object Motion Blur"}],"recommendation":"major_revision","confidential_remarks":"This is a substantial benchmark contribution, and the released data alone will likely be useful to the community. The main weakness is that several of the headline observations, especially those about GoPro-FC and the VFM benefit, are not yet supported by the evidence as analyzed. I would encourage the authors to add the small-displacement RCRE validation and to de-confound the SAMFlow comparison; these are additional analyses rather than a change of scope, so I believe major revision is the appropriate outcome."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Main take: this is a genuinely useful benchmark paper, the first to systematically adapt corruption robustness benchmarking to optical flow, and it ships the right kind of artifact: corrupted image pairs, code, and a large evaluation matrix (29 model variants, 24 corruptions). If I worked on optical flow robustness, I'd want this resource even before the conclusions.\n\nWhat's new and good: the 7 temporal corruptions tailored to flow (exposure mismatch between frames, object/camera motion blur, H.264 compression, bit errors) fill a real gap in the single-image corruption literature. The PSF blur from real lens designs is a step up from generic blur kernels. The evaluation is careful: same training pipeline, official checkpoints, fixed decoder iterations, consistent image sizes. That is reproducible work and it shows.\n\nSoft spots, in proportion. The biggest is GoPro-FC. Without ground truth, all quantitative claims rest on RCRE, which measures prediction shift between clean and corrupted outputs. The paper validates RCRE against CRE only on KITTI-FC, a large-displacement benchmark with ground truth; there is no evidence that RCRE rankings match true EPE-based robustness in the small-displacement regime of GoPro-FC. Observations 9 and 10 are the weakest because they depend on that unvalidated proxy. I'd want either a rank-correlation check on a small-displacement subset with ground truth or explicit hedging.\n\nSecond, the weather corruptions apply the identical spatter/fog/frost texture to both frames. The paper justifies it, and it's defensible, but it can flatter correlation-based models that lock onto static texture. An ablation with independent noise samples would tighten the claim. Third, Observation 1 is partly circular since CRE is an EPE difference; the appendix acknowledges the linear relationship, but the 'heavily dependent' phrasing overstates what the metric can show. That's minor. The SAMFlow vs FlowFormer comparison is also confounded by training steps and the SAM module; the VFM conclusion should be softened.\n\nInconsistent counts (10 vs 7 observations) and missing error bars are minor editorial issues.\n\nWho it's for: anyone building or evaluating optical flow models for driving or video editing. The benchmark and code will likely be adopted. It deserves a serious referee. I'd engage with it, but I'd want the GoPro-FC quantitative claims either validated or heavily qualified before trusting them.","headline":"The first real corruption-robustness benchmark for optical flow, with a careful large-scale evaluation; the GoPro-FC results should be read as provisional because they rest on an unvalidated no-ground-truth proxy.","tokens_in":30210,"tokens_out":2090,"would_cite":true,"duration_ms":21336,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims to build the first corruption robustness benchmark for optical flow estimation, covering 24 corruptions and 29 model variants on KITTI-FC and GoPro-FC.","keywords":["optical flow estimation","corruption robustness benchmark","temporal corruptions","KITTI-FC","GoPro-FC","robustness metrics","out-of-domain evaluation","unsupervised optical flow"],"falsifier":"Corrupt only one frame of each pair, or use independently sampled per-frame spatter, frost, and fog overlays, then rerun RAFT, FlowFormer, and SAMFlow on KITTI-FC and compare the weather-robustness ranking with the shared-overlay setting; if the ranking changes materially, the benchmark's weather robustness numbers are an artifact of the identical-perturbation assumption.","tokens_in":29211,"feed_emoji":"🎥","tokens_out":4619,"duration_ms":44752,"temperature":0.7,"pith_summary":"The paper sets out to give optical flow estimation its first systematic way to measure how models behave when inputs are corrupted in ways common in real-world use, rather than only adversarially perturbed. It builds two benchmarks, KITTI-FC and GoPro-FC, spanning 24 corruptions, including seven temporal corruptions that exploit the two-frame nature of optical flow. Across 29 model variants it finds that a model's absolute robustness is largely determined by its clean-data accuracy, that corruptions destroying local information hurt more than mere visual degradation, and that unsupervised models are relatively stable. If the findings hold, robustness can be improved mainly by improving estimation performance, by using transformer-like and semantic features, and by preferring unsupervised training.","feed_headline":"24 corruptions, 29 models: a robustness test for optical flow","feed_subtitle":"KITTI-FC and GoPro-FC show absolute robustness tracks clean accuracy, and unsupervised models lead relative robustness.","key_machinery":"The central mechanism is the benchmark itself, built from a corruption suite and paired evaluation protocols. The suite contains 24 corruptions in six classes, including seven temporal corruptions (Over Exposure, Under Exposure, Object Motion Blur, Snow, H.264 CRF, H.264 ABR, and Bit Error) that alter the two frames asymmetrically, alongside 17 single-image corruptions, with an upgraded PSF Blur generated from five simulated low-quality lenses. The benchmarks KITTI-FC and GoPro-FC respectively emphasize large and small displacements, and the metrics CRE, CREr, and RCRE quantify absolute and relative robustness, with RCRE usable when ground-truth flow is unavailable.","core_discovery":"The paper's central claim is that optical flow robustness to common corruptions can and should be benchmarked systematically, and that the field has lacked such a benchmark. To support this, the authors introduce 24 corruptions split into six classes, with seven temporal corruptions specifically designed for image pairs, and construct KITTI-FC and GoPro-FC for autonomous driving and video editing scenarios, each with Out-Of-Domain and In-Domain settings. They propose Corruption Robustness Error (CRE), Corruption Robustness Error ratio (CREr), and Relative Corruption Robustness Error (RCRE) as metrics. Evaluating 29 model variants from 15 methods, they report ten observations, most notably that absolute robustness is heavily dependent on estimation performance, that corruptions which destroy local information are more serious than those which only reduce visual effects, and that unsupervised methods show strong relative robustness.","pith_inferences":["A per-frame independent rendering of weather corruptions would be a stricter test; correlation-based models might match static spatter or frost patterns and appear more robust than a truly dynamic scene would allow.","Because CRE and RCRE are almost linearly related in the paper's own data, RCRE could stand in for CRE on any real video without ground truth, widening the benchmark to unlabeled footage.","The small-displacement failure of SAMFlow suggests semantic segmentation guidance is motion-scale dependent; finer or motion-aware masks could recover the benefit in GoPro-like settings.","Composite corruptions, which the paper lists as out of scope, are the natural next stress test and could be generated by composing the released per-corruption pipelines."],"forward_implications":["Future optical flow evaluations can report CRE, CREr, and RCRE on KITTI-FC and GoPro-FC alongside EPE, making corruption robustness a standard comparison axis.","Improving clean-data estimation accuracy is a direct route to better absolute robustness, so progress on clean benchmarks should transfer to corrupted conditions.","Transformer-like architectures and semantic information help under large-displacement corruptions, but the benefit does not carry to small-displacement settings.","Unsupervised models are a practical robustness strategy: ARFlow achieves the best relative robustness with the smallest computing cost.","Fine-tuning to an in-domain target improves absolute robustness but lowers relative robustness, with weather corruptions becoming relatively more destabilizing."],"supporting_citations":[{"why":"Supplies the single-image corruption paradigm and many of the classical corruptions that the paper extends to optical flow.","marker":"[14]"},{"why":"Source of the weather corruption rendering used for Spatter, Fog, Frost, and Snow with severity levels.","marker":"[46]"},{"why":"Provides the H.264 CRF compression corruption and the video-oriented benchmarking approach adopted for temporal corruptions.","marker":"[78]"},{"why":"KITTI optical flow ground-truth data forms the base of KITTI-FC.","marker":"[45]"},{"why":"GoPro dataset forms the base of GoPro-FC for the video-editing scenario.","marker":"[49]"},{"why":"FLAVR video interpolation enables the 4x interpolation needed for Object Motion Blur and the video corruptions in GoPro-FC.","marker":"[21]"},{"why":"GSO automatic lens design generates the five PSF blur kernels used in the upgraded PSF Blur corruption.","marker":"[12]"},{"why":"RAFT supplies the common training pipeline for supervised models and serves as a central baseline.","marker":"[70]"},{"why":"SAMFlow provides the Vision Foundation Model variant behind the observations on semantic information and VFM usefulness.","marker":"[89]"}],"fun_headline_variants":["First benchmark for optical flow robustness to corruptions","Optical flow robustness: 24 corruptions, 29 models tested","Absolute robustness tracks clean accuracy in optical flow","Unsupervised optical flow models lead in relative robustness","KITTI-FC and GoPro-FC: new robustness benchmarks for optical flow"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark assumes that weather effects such as spatter, fog, and frost affect both frames of an image pair identically, applying the same corrupted rendering to both frames; if real weather artifacts differ subtly between frames, the measured weather robustness could overstate how well models handle true weather.","fun_headline_variants_meta":{"raw":{"variants":["First benchmark for optical flow robustness to corruptions","Optical flow robustness: 24 corruptions, 29 models tested","Absolute robustness tracks clean accuracy in optical flow","Unsupervised optical flow models lead in relative robustness","KITTI-FC and GoPro-FC: new robustness benchmarks for optical flow"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000625,"raw_usage":{"total_tokens":2934,"prompt_tokens":1026,"completion_tokens":1908,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":642,"completion_tokens_details":{"reasoning_tokens":1824}},"tokens_in":642,"tokens_out":1908,"duration_ms":13138,"temperature":1.0,"reasoning_tokens":1824,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:46:55.597767+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Corrupt only one frame of each pair, or use independently sampled per-frame spatter, frost, and fog overlays, then rerun RAFT, FlowFormer, and SAMFlow on KITTI-FC and compare the weather-robustness ranking with the shared-overlay setting; if the ranking changes materially, the benchmark's weather robustness numbers are an artifact of the identical-perturbation assumption.","supporting_citations":[{"cited_title":"Dietterich","cited_arxiv_id":null,"evidence_quote":"Supplies the single-image corruption paradigm and many of the classical corruptions that the paper extends to optical flow."},{"cited_title":"Benchmarking the robustness of spatial-temporal models against corruptions","cited_arxiv_id":null,"evidence_quote":"Provides the H.264 CRF compression corruption and the video-oriented benchmarking approach adopted for temporal corruptions."},{"cited_title":"Object scene flow for autonomous vehicles","cited_arxiv_id":null,"evidence_quote":"KITTI optical flow ground-truth data forms the base of KITTI-FC."},{"cited_title":"Deep multi-scale convolutional neural network for dynamic scene deblurring","cited_arxiv_id":null,"evidence_quote":"GoPro dataset forms the base of GoPro-FC for the video-editing scenario."},{"cited_title":"FLA VR: Flow-agnostic video representations for fast frame interpolation","cited_arxiv_id":null,"evidence_quote":"FLAVR video interpolation enables the 4x interpolation needed for Object Motion Blur and the video corruptions in GoPro-FC."},{"cited_title":"RAFT: Recurrent all-pairs field transforms for optical flow","cited_arxiv_id":null,"evidence_quote":"RAFT supplies the common training pipeline for supervised models and serves as a central baseline."},{"cited_title":"SAMFlow: Eliminating any fragmentation in optical flow with segment anything model","cited_arxiv_id":null,"evidence_quote":"SAMFlow provides the Vision Foundation Model variant behind the observations on semantic information and VFM usefulness."}],"review_version":1}