{"id":"a487b5e2-9d39-44aa-b63e-f79afd2dfcbf","arxiv_id":"2411.16926","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An adaptive input-composition rule for video inpainting uses optical flow and mask changes to swap reference frames for neighboring frames, improving quality on fast-moving scenes.","lead":"This paper studies how the mix of nearby and distant frames fed to a video inpainting program affects output quality, and proposes a rule for adjusting that mix based on how much the scene is moving. If the rule works, it could help phones and AR glasses fill in missing video regions while using less memory.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The adaptive composition rule is fit and evaluated on the same DAVIS videos, so the claimed gain may be overfitting rather than a general property; a held-out split is needed before the central claim is supported.","rationale":"The central claim is that dynamically adjusting input composition improves video inpainting under a fixed memory budget. For that claim to hold, the slopes, bin boundaries, and assigned reference-frame ratios in Section 4.2 must generalize beyond the videos on which they were estimated. The paper's own limitation statement confirms the configuration is heuristic, but the more serious issue is the evaluation protocol: the same DAVIS data used to fit the heuristic is used to evaluate it, with no held-out split or error bars. The reported gains are small (e.g., +0.1092 dB overall on DAVIS with ProPainter) and MOSE is negative (-0.0591 dB), so the evidence is too weak to support a general claim. The reader's weakest assumption identifies the same concern, so I agree. If a clean cross-validation or held-out evaluation shows consistent positive gains on both datasets, the claim would become credible; as it stands, it remains conditional on that missing evidence.","tokens_in":9778,"tokens_out":5934,"duration_ms":59124,"concrete_test":"Run leave-one-video-out cross-validation on the DAVIS and MOSE test sets: for each held-out video, fit m_flow, m_mask, the bin boundaries, and the reference-frame ratio assignments using exactly the Section 4.2 procedure on all other videos; then run AdaptIn with ProPainter on the held-out video and compare PSNR/SSIM/VFID against the fixed 5:5 baseline. Report the mean and bootstrap 95% CI of the held-out differences for each dataset. If the mean gain is not positive with a CI excluding zero, the fitted heuristic does not generalize; as a secondary control, shuffle the ratio assignments across bins and verify that the real assignment beats shuffled assignments by more than selection noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 builds AdaptIn on the regression slopes m_flow = -0.1644 and m_mask = -0.1285 (Figure 4), and then chooses the seven bin boundaries and the reference-frame ratios (0.875 down to 0.125 in Figure 7) from the observed trend. These estimates come from the DAVIS 2017 samples analyzed in Section 3.2, and Section 5 then evaluates the same pipeline on DAVIS plus MOSE. The DAVIS evaluation is therefore in-sample, and the heuristic is a fit to the same videos whose performance is being reported. The scatter is large (RSS 0.5062 for flow, 0.6033 for mask, 0.4586 for the combined fit), so the negative trend is weak and leaves most variance unexplained. Because the mapping is selected to match this particular data, the positive mean PSNR changes in Tables 1 and 2 do not demonstrate that AdaptIn generalizes; a random or arbitrary assignment of reference ratios to bins could plausibly perform similarly under the same selection procedure. The paper explicitly labels the configuration heuristic in Section 6, but the decisive problem is not that it is heuristic: it is that the heuristic is fit and evaluated without a held-out split. MOSE is a partial out-of-sample check, but Table 1 reports a negative overall PSNR change on MOSE (-0.0591), so the transfer is not established either.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AdaptIn, a preprocessing pipeline that dynamically configures the input composition for video inpainting models by adjusting the ratio of reference frames to neighboring frames in an 8-frame window. The configuration is driven by visual dynamics measured through completed optical flow in the masked region and temporal mask variation. The authors first present observational evidence (§3) that inpainting quality varies with this ratio and that the direction of the preferred ratio correlates with flow and mask changes (Figures 4, 5, 7). They then describe the AdaptIn pipeline (§4), which fits linear slopes to these correlations, computes a combined variable xcomb, partitions its range into seven bins, and assigns each bin a reference-frame ratio between 0.125 and 0.875. Experiments on DAVIS 2017 and MOSE (§5) with ProPainter and STTN are reported as improvements in PSNR, SSIM, and VFID relative to a balanced (5:5) baseline, together with qualitative examples and a memory-quality trade-off analysis.","tokens_in":10146,"tokens_out":2536,"duration_ms":25496,"significance":"If validated, the paper's core observation—that the optimal input frame composition depends on local visual dynamics—could be practically relevant for memory-constrained video inpainting on mobile or AR devices. The manuscript makes a useful conceptual point and offers a simple, potentially efficient heuristic. However, the currently reported evidence is not strong enough to establish the claimed improvements as general: the effect sizes are small, the fitted configuration is evaluated on the same data that produced it, and the out-of-sample dataset shows a negative overall PSNR change for the primary model. The paper does include machine-checked experiments in the sense of reproducible tables and figures, but it lacks a held-out validation protocol. The honest limitations section is a positive feature, but it also confirms that the central heuristic is unvalidated.","major_comments":[{"comment":"The evaluation of AdaptIn is in-sample for the DAVIS results. The linear slopes (m_flow = -0.1644, m_mask = -0.1285) in Figure 4, the combined slope in Figure 7, and the seven bin boundaries with the assigned reference-frame ratios are all derived from the same DAVIS 2017 samples on which Table 1 and Table 2 later report improvements. Because the bin-to-ratio mapping is selected to match this data, the positive mean PSNR changes for DAVIS do not demonstrate generalization. The only out-of-sample set, MOSE, actually shows a negative overall PSNR change for the ProPainter baseline in Table 1 (-0.0591). The paper needs a held-out split, cross-validation, or a genuine train/evaluation separation before the central claim that AdaptIn improves quality can be supported.","section":"§4.2, §5, Figure 7, Table 1"},{"comment":"The linear trends that motivate the entire configuration are weak and exhibit high residual variance. RSS values of 0.5062 (flow), 0.6033 (mask), and 0.4586 (combined) indicate that most of the variance in the maximum change rate remains unexplained, and the slopes are small relative to the scatter. Given this noise, the specific seven-bin partition and the assigned reference-frame ratios in Figure 7 are underdetermined. A random or arbitrary assignment of ratios to bins under the same selection procedure could plausibly produce similar mean PSNR differences, especially when the per-bin differences in Tables 1 and 2 are on the order of 0.1 dB or less. The authors should provide confidence intervals, significance tests, or a permutation-based baseline to show that their bin assignment is not an artifact of the fitting procedure.","section":"§3.2, Figures 4 and 7"},{"comment":"The reported improvements are numerically very small, and the manuscript does not establish that they are statistically meaningful. The overall PSNR gains are about 0.1 dB on DAVIS, the SSIM differences are reported in units of 10^-5, and the VFID differences are in units of 10^-3. These magnitudes are close to typical metric noise, and no error bars, per-sequence variances, or significance tests are provided. The qualitative statement that 'performance variations were more accurately captured by the VFID metric' (§5.1.1) is not supported by any quantitative comparison across metrics. The authors should report variability over the 45 and 171 videos and test whether the 'Overall' row differs significantly from zero.","section":"§5, Tables 1 and 2"}],"minor_comments":[{"comment":"The Signed Maximum Change Rate in Eq. (1) is not clearly defined in terms of the sampling of r. The set P is given as r in [0.125, 0.25, ..., 0.875], but the text later refers to intervals of 0 or more and below 0; please clarify whether the max/min in Eq. (1) are taken over the discrete set P, and define how the sign is computed when the maximum and minimum occur at multiple r.","section":"§3.2, Eq. (1)"},{"comment":"The notation is inconsistent between x_comb in the text and xcomp in Figure 7; also, x_f low and x_mask are defined only verbally. Please introduce explicit normalized feature definitions with the same scales as in Figure 4.","section":"§4.1, §4.2"},{"comment":"The table caption states that r_ref and r_nei are ratios, and the baseline is r_ref=0.5, r_nei=0.5. However, the rows list r_nei values such as 0.125, 0.25, 0.325, etc. It would be clearer to state that the tables report differences relative to the baseline, and to include the baseline row explicitly with zero entries rather than '-'. Also, the value 0.325 for 'rnei' should be 0.375 to be consistent with the set in §3.2; this appears to be a typo in both tables.","section":"§5.1.1, Tables 1 and 2"},{"comment":"The memory-quality tradeoff is claimed to vary the number of input frames from 11 to 5, but Figure 9's x-axis shows memory usage from 4 to 8 GB without a clear mapping to the number of frames. Please specify the frame-count-to-memory relationship and indicate where the 8-frame constraint of the main experiments lies on this axis.","section":"§5.2, Figure 9"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is plausible and the authors are honest about the heuristic nature of their configuration, which is a positive sign. However, the current evidence is not sufficient to support the general claim: the central heuristic is fit and evaluated on the same DAVIS data, the MOSE transfer test shows a negative effect for the main model, and the effect sizes are extremely small. A major revision that adds a proper held-out evaluation (e.g., fitting on one dataset and evaluating on another, or cross-validation on DAVIS) and reports variability/statistical significance could make the paper acceptable. The manuscript also has some presentation issues that should be fixed in the same revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Seems worth a referee's time, but the central quantitative claim is not yet supported by the evidence as presented. The paper does something genuinely useful: it measures how the ratio of reference to neighboring input frames affects video inpainting quality under a fixed 8-frame memory budget, across DAVIS, and finds a consistent (though noisy) tendency: faster optical flow and larger mask changes favor neighboring frames. That correlation study is the real contribution, and it is presented clearly.\n\nAdaptIn itself is a simple, sensible heuristic built on that observation, and the paper is honest that the configuration is heuristic (Section 6). It also reuses ProPainter's flow completion, which is a practical efficiency choice.\n\nThe soft spot is exactly where you'd expect: the regression slopes and the seven-bin mapping are fit on DAVIS and then evaluated on DAVIS. The reported mean PSNR gains are about 0.1 dB, the fits have RSS around 0.5, and on MOSE the overall PSNR change is negative for ProPainter. So the in-sample improvement is plausibly partly overfitting to the very videos used to set the bins. The paper needs a held-out split, cross-validation, or a fixed rule derived from a separate validation set, plus error bars across video samples. Without that, the tables do not demonstrate generalization.\n\nThat said, the paper is honest about its limitations, and the qualitative object-removal results are suggestive. The central intuition—neighboring frames matter more in fast scenes—is well known in video processing; the paper's contribution is quantifying it in this specific inpainting setting and showing that a cheap input-composition policy can move quality a little. That is modest but real.\n\nWho is this for? Someone working on memory-constrained video inpainting on devices, or on input sampling strategies for transformers. A serious referee could push for a cleaner evaluation and the paper would be stronger for it. I would not desk-reject; the correlation study alone justifies a round of review.\n\nRecommendation: send to peer review, but tell the authors the evaluation needs a held-out split and explicit error bars before the central claim can be taken at face value.","headline":"A useful correlation study and a plausible heuristic, but the central quantitative claim is not yet convincing because the input-configuration rule is fit and evaluated on the same videos.","tokens_in":10646,"tokens_out":1627,"would_cite":false,"duration_ms":15293,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that the best mix of neighboring and reference frames for video inpainting depends on optical flow and mask motion, and that a context-aware input configurator improves quality under a fixed 8-frame memory budget.","keywords":["video inpainting","input frame composition","neighboring frames","reference frames","optical flow","mask change","memory-constrained inference"],"falsifier":"On a held-out set of videos with fast motion, run AdaptIn and the balanced 5:5 baseline under the same 8-frame budget; if AdaptIn's chosen configuration does not beat the baseline in PSNR or VFID for clips whose x_comb falls into the three neighboring-dominant bins, the fitted mapping is not generalizing.","tokens_in":9618,"feed_emoji":"🎬","tokens_out":4874,"duration_ms":41281,"temperature":0.7,"pith_summary":"Video inpainting models usually receive a fixed mix of neighboring frames and distant reference frames. This paper argues that the best mix depends on how fast the visual content is moving: when optical flow and mask motion are large, neighboring frames matter more than reference frames, and when the scene is static, reference frames help more. It confirms this correlation on standard video-inpainting benchmarks and builds a pipeline, AdaptIn, that measures optical flow and mask change for each target frame and then picks one of seven input configurations under a fixed 8-frame memory budget. The payoff is a memory-quality tradeoff: at the same memory cost, dynamically configured inputs improve inpainting quality, especially in fast-motion sequences and for flow-guided inpainting backbones.","feed_headline":"Adaptive frame mix lifts video inpainting under 8-frame budget","feed_subtitle":"Choosing neighboring vs reference frames by motion and mask change preserves quality when memory caps inputs at 8 frames.","key_machinery":"The central machinery is AdaptIn, a preprocessing pipeline with three parts: a mask-change analyzer that sums pixel-wise differences between consecutive masks, an optical-flow analyzer that completes flow in masked regions using the same recurrent module as ProPainter, and an input configurator that combines the two into x_comb using the fitted regression slopes m_flow = -0.1644 and m_mask = -0.1285 as weights. The configurator maps x_comb to one of seven bins, each prescribing a fixed ratio of reference to neighboring frames in the 8-frame input set. The key identity is the linear relation between x_comb and the maximum PSNR change rate, whose slope and bins are fitted on DAVIS and MOSE data.","core_discovery":"The paper discovers that the influence of an input frame on inpainting quality is not uniform; it depends on visual dynamics in the masked region. Using a signed maximum change rate in PSNR, it shows that as normalized optical flow and mask change increase, the optimal input shifts toward neighboring frames, with a linear fit slope of -0.1644 for optical flow and -0.1285 for mask change. Combining these two signals into a weighted factor x_comb yields a steeper linear trend (slope -0.2864, RSS 0.4586), which AdaptIn partitions into seven ranges, each mapped to a reference-frame ratio from 0.125 to 0.875. The paper claims this context-aware composition improves inpainting quality over the balanced 5:5 baseline under the same 8-frame memory limit, and that the flow-guided inpainter ProPainter benefits more than the transformer STTN.","pith_inferences":["The same input-orchestration idea could transfer to other video restoration tasks such as super-resolution, deblurring, or frame interpolation, where a similar reference-versus-neighbor tradeoff exists, but the paper does not test those settings.","The hand-fitted regression slopes and seven bin boundaries could be replaced by a learned mapping from flow and mask statistics to input composition, which might generalize better than the paper's heuristic rule.","The paper's object-removal claim rests on qualitative comparisons; a dedicated temporal-consistency metric for removed regions would be needed to verify the claimed coherence gain numerically."],"forward_implications":["Under a fixed memory budget, inpainting quality can be improved by changing which frames enter the model, rather than by adding more frames.","Flow-guided inpainting models, which already compute completed flow, can reuse that flow for input selection at almost no extra cost.","Object removal benefits from context-aware composition: static backgrounds borrow features from reference frames, dynamic regions borrow from neighbors.","The benefit concentrates in fast-motion data; datasets like DAVIS with more dynamic clips show larger overall gains than MOSE.","When memory is cut to the smallest budgets, the adaptive composition's quality gap over the balanced baseline narrows, because flow-guided propagation already supplies much of the benefit."],"supporting_citations":[{"why":"Supplies the ProPainter baseline, its flow completion module, and the comparison that drives the central claim.","marker":"[6]"},{"why":"Provides STTN, the transformer baseline used to show that even non-flow models are influenced by input composition.","marker":"[3]"},{"why":"DAVIS 2017 is the primary benchmark dataset used both for fitting the regression and for evaluating quality gains.","marker":"[7]"},{"why":"RAFT is the optical flow model used to compute flow between streamed frames before completion.","marker":"[10]"},{"why":"MOSE is the second evaluation dataset, used to test generalization of the fitted configuration across motion distributions.","marker":"[12]"},{"why":"BasicVSR++ supplies the second-order deformable alignment used by ProPainter's flow completion module.","marker":"[11]"}],"fun_headline_variants":["Video inpainting adapts frame mix to motion and mask","Flow and mask guide frame choice for inpainting","Context-aware frame selection boosts inpainting quality","Motion-aware input orchestration improves video inpainting","Adaptive frame ratios enhance inpainting under 8-frame limit"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The seven bin boundaries and the two regression slopes that drive frame selection are fitted on the same DAVIS and MOSE videos used to report quality gains, and the paper assumes they keep working on unseen videos and with other flow completion modules.","fun_headline_variants_meta":{"raw":{"variants":["Video inpainting adapts frame mix to motion and mask","Flow and mask guide frame choice for inpainting","Context-aware frame selection boosts inpainting quality","Motion-aware input orchestration improves video inpainting","Adaptive frame ratios enhance inpainting under 8-frame limit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000234,"raw_usage":{"total_tokens":1436,"prompt_tokens":826,"completion_tokens":610,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":442,"completion_tokens_details":{"reasoning_tokens":530}},"tokens_in":442,"tokens_out":610,"duration_ms":5173,"temperature":1.0,"reasoning_tokens":530,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:45:00.060141+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out set of videos with fast motion, run AdaptIn and the balanced 5:5 baseline under the same 8-frame budget; if AdaptIn's chosen configuration does not beat the baseline in PSNR or VFID for clips whose x_comb falls into the three neighboring-dominant bins, the fitted mapping is not generalizing.","supporting_citations":[{"cited_title":"ProPainter: Improving propagation and transformer for video inpainting","cited_arxiv_id":null,"evidence_quote":"Supplies the ProPainter baseline, its flow completion module, and the comparison that drives the central claim."},{"cited_title":"Learning joint spatial-temporal transformations for video inpainting","cited_arxiv_id":null,"evidence_quote":"Provides STTN, the transformer baseline used to show that even non-flow models are influenced by input composition."},{"cited_title":"Raft: Recurrent all-pairs field transforms for optical flow","cited_arxiv_id":null,"evidence_quote":"RAFT is the optical flow model used to compute flow between streamed frames before completion."},{"cited_title":"Chan, Shangchen Zhou, Xiangyu Xu, and Chen Change Loy","cited_arxiv_id":null,"evidence_quote":"BasicVSR++ supplies the second-order deformable alignment used by ProPainter's flow completion module."}],"review_version":1}