{"id":"9178676a-b58b-4af5-89d5-0a11031866a7","arxiv_id":"2509.07472","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A training-free video background replacement pipeline that keeps the foreground pixel-consistent by projecting refined latents through a deterministic reparameterization.","lead":"ANYPORTAL replaces video backgrounds without training by combining a video diffusion model, IC-Light relighting, and a deterministic latent projection (RPA) that preserves foreground details. It runs on one 24GB GPU and beats several zero-shot baselines in user studies, but its code and test data are not yet public.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RPA's epsilon-transfer assumption is unvalidated: when refinement changes the VAE posterior, reusing the unrefined epsilon can inject background drift; identity preservation is proven only for the no-refinement case.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the deterministic epsilon transfer in Algorithm 2. The paper's key invariant is exact only when no refinement is applied, which never occurs in actual use; when Refine is applied, the reparameterization noise epsilon_hat computed from the unrefined decode may not be a valid reparameterization of the refined latent. This is a genuine soft spot in the central claim, not merely a matter of evidence quality. However, the concern does not by itself invalidate the approach: a targeted measurement of background drift could show that the projection's error is negligible in practice. The paper's empirical results are suggestive but limited (30 samples, no code/data release, no error bars, no isolated background-drift metric), so the correct verdict remains CONDITIONAL as the reader concluded. No formal verification or released code provides independent support, and the paper's own Limitations section acknowledges boundary and motion artifacts, further motivating a quantitative check of RPA's projection behavior.","tokens_in":12408,"tokens_out":8053,"duration_ms":88671,"concrete_test":"Implement Algorithm 2 on a set of at least 10 diverse input videos (or use the paper's reported 30-sample set once released). For each timestep t, record I_t^0 = D(x_t^0), the refinement tilde_I_t^0 = Refine(I_t^0, I), and the projected latent x_hat_t^0. Compute the background-only decoded drift Delta_t = D(x_hat_t^0) - I_t^0 masked by (1 - M_t^0), and compare its mean squared magnitude to (a) the foreground refinement magnitude tilde_I_t^0 - I_t^0 masked by M_t^0, and (b) the VAE reconstruction error D(E(I_t^0)) - I_t^0. If ||Delta_t|| on background is close to (b) and much smaller than (a), the epsilon-transfer assumption is validated. If it approaches (a) in background regions, the claim 'background area remains almost identical' fails and the central foreground-consistency mechanism needs revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Sec. 3.2.3/Algorithm 2, RPA's central property is stated as: if no refinement is applied, the projection returns x_hat_t^0 = x_t^0 exactly. This is true by algebra: epsilon_hat = (x_t^0 - mu)/sigma and x_hat_t^0 = mu + epsilon_hat * sigma = x_t^0. But the actual pipeline always applies Refine, and the projection reuses epsilon_hat computed from the unrefined decode to reparameterize E(Refine(D(x_t^0), I)) = (mu_hat, sigma_hat). Since E is a nonlinear 3D VAE and Refine alters foreground high-frequency details and inpaints the background, mu_hat and sigma_hat can change nonlocally; nothing guarantees the normalized deviation transfers. The paper's support is only the qualitative Fig. 5(d) and the end-to-end ablation in Table 2, which does not isolate whether the background remains 'almost identical' as claimed. No code or data are released, so the zero-error projection's behavior under refinement is unverified. If sigma_hat differs from sigma in background regions, x_hat_t^0 differs from x_t^0 there, injecting background drift that accumulates over T1 denoising steps.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"ANYPORTAL is a training-free pipeline for video background replacement from an input foreground video plus a text/image background prompt. It consists of three stages: motion-aware background video generation using DAS/ControlNet and ProPainter; a two-step IC-Light illumination harmonization with cross-frame attention; and a consistency-enhancement stage that runs SDEdit denoising with a frozen video diffusion model while injecting high-frequency foreground details through a Refinement Projection Algorithm (RPA). RPA decodes the current latent, refines foreground details in pixel space, re-encodes the refined video, and reparameterizes the refined latent using a deterministic noise computed from the unrefined decode. The authors prove an exact-reconstruction identity when no refinement is applied and claim this keeps background regions almost identical during actual refinement. Experiments compare against IC-Light, TokenFlow, and DAS on 30 self-collected videos, with additional ablations.","tokens_in":12683,"tokens_out":7213,"duration_ms":81173,"significance":"RPA is an elegant attempt to make pixel-level foreground control compatible with compact 3D video latents, and the no-refinement identity is exactly stated and mathematically correct. The modular, fully frozen-model design is practical and clearly described; the limitations section is honest. However, the main technical novelty is only validated qualitatively under the actual refinement operation, and the empirical evidence is small-scale with no statistical significance. If the stability of RPA under refinement is demonstrated with quantitative background-preservation metrics and the code/data are released, this would be a useful contribution to zero-shot video editing. As written, the strengths are not yet matched by the evidence.","major_comments":[{"comment":"The exact-reconstruction property is shown only for the no-refinement case (tilde I_t^0 = I_t^0). In the actual pipeline, Refine always changes the decoded video, so the reparameterization epsilon_hat=(x_t^0-mu)/sigma is reused with (mu_hat,sigma_hat)=E(Refine(D(x_t^0),I)). If the nonlinear VAE and the inpainted background shift mu_hat/sigma_hat outside the edited foreground, hat x_t^0 = mu_hat + epsilon_hat sigma_hat accumulates drift in every DDIM step. The paper does not analyze this transfer assumption or quantify background drift; Fig. 5(d) and Table 2 are not isolate it. Please add a background-masked distortion comparison (RPA vs. no-RPA, or vs. direct encode-decode) across T1 values and refinement strengths, and state conditions under which the transfer is stable.","section":"Sec. 3.2.3 / Algorithm 2"},{"comment":"The evaluation is based on 30 self-collected samples; no standard deviations, confidence intervals, or significance tests are reported. Several differences are small (e.g., Tem-Con 0.993 vs 0.986 for DAS; Mtn-Psrv 0.987 vs 0.985; ID-Psrv 0.313 vs 0.364) and may lie within noise. The user study uses 24 raters but only preference percentages are given. Without per-sample variability or paired tests, the claim that ANYPORTAL outperforms the baselines is not statistically supported. Code and test data are also promised only for publication, preventing independent verification.","section":"Sec. 4 / Table 1"},{"comment":"The description and the pseudocode disagree on the inpainting target. The text states the background region of tilde I_t^0 is set to 'the inpainted I', while Algorithm 1 line 6 computes I_BG = Inpaint(I_t^0, ForegroundSegmentation(I_t^0)) and line 7 uses it with the mask M_t^0 computed from I. Inpainting the edited decode versus the original input are different operations, and using a mask from one video to select an inpainted region from another can misalign boundaries. Please clarify the intended operation and align the text with the implementation; this choice directly bears on the claim that background areas are preserved by RPA.","section":"Sec. 3.2.3 / Algorithm 1"},{"comment":"The ablation for RPA is ambiguous. The text compares 'without RPA' (Fig. 5(b)) to 'naive high-frequency detail refinement through decoding and encoding' (Fig. 5(c)), but Table 2 only has a single 'w/o RPA' row. It is unclear whether w/o RPA means no foreground refinement at all, or refinement by direct encode-decode without the deterministic projection. These are different ablations; the former does not isolate RPA and the latter does. Please specify the exact configuration and, ideally, report both variants.","section":"Sec. 4.2 / Table 2"}],"minor_comments":[{"comment":"Notation is inconsistent: the text uses epsilon_hat while the pseudocode uses \\hat\\epsilon; also make explicit in Algorithm 2 that x_T1 is obtained by the DDPM forward process from I_L, since only 'Initial noise x_T1' is stated.","section":"Algorithm 2"},{"comment":"The sentence 'for each frame of I\\'' refers to an output not yet defined; clarify whether it is I from the input video or the background-generation output.","section":"Sec. 3.2.2"},{"comment":"The comparison with Light-A-Video is qualitative only. If quantitative metrics are available for this concurrent work, include them; otherwise, state explicitly that no metric was computed.","section":"Fig. 10"},{"comment":"The limitations section is commendable, but it would be useful to also state the failure mode of RPA when the foreground-background boundary segmentation is inaccurate, since the refinement directly depends on the mask quality.","section":"Sec. 6"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a straightforward systems paper whose main novelty is the RPA projection. The cross-period claims and the central RPA stability argument need additional quantitative support before acceptance. No citation or scope concerns. I would favor a major revision with targeted experiments on background drift, a corrected Algorithm 1, and clearer ablations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: the Refinement Projection Algorithm is a neat, correctly-stated deterministic reparameterization trick, and the modular pipeline (IC-Light + DAS + CogVideoX + cross-frame attention) is genuinely training-free and runs on a 24GB GPU. The algebra behind RPA is right: given the unrefined decode and its VAE statistics, you can solve for a deterministic epsilon and reuse it with the refined latent to avoid random re-encoding drift. That's a real contribution. The qualitative figures are convincing, and the ablation is sensibly designed.\n\nThe soft spots: the evidence is thin. Thirty self-collected samples, no error bars, no significance tests, no code or data. The user study is small but directionally positive. There's also a discrepancy between the text and Algorithm 1: the text says the background is the inpainted original input I, but Algorithm 1 inpaints I_t0. That should be fixed. More importantly, the RPA identity property is proven only for the no-refinement case, and the pipeline always refines. The stress-test worry about epsilon-transfer is legitimate: if E is nonlinear, mu_hat and sigma_hat can shift in background regions, and reusing the old epsilon can inject drift. The paper doesn't analyze when this breaks, and Fig. 5(d) alone doesn't settle it. That's not a fatal flaw—the examples look good—but it's a missing analysis. The claims about 'almost identical' backgrounds outrun the proof.\n\nBottom line: this is a practical engineering paper, not a scientific breakthrough. For a reader working on video editing or diffusion-based control, the RPA trick is worth knowing. The authors need to release code and data, correct the algorithm/text inconsistency, and add a stability check on RPA under refinement. With that, it's a solid contribution. I'd send it to peer review.","headline":"ANYPORTAL is a clever training-free pipeline with a genuine new trick in RPA, but the evidence base is too thin for the strength of the claims; still, it deserves a serious referee.","tokens_in":13198,"tokens_out":4149,"would_cite":false,"duration_ms":45810,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A training-free pipeline replaces video backgrounds while preserving foreground pixels.","keywords":["video background replacement","zero-shot video editing","diffusion models","foreground consistency","relighting","latent space projection","temporal consistency","training-free pipeline"],"falsifier":"Take a video with fast motion or a low-resolution foreground, run AnyPortal with RPA active and with RPA replaced by naive decode-refine-encode, and compare the background regions pixel-by-pixel against the no-refinement output. If the background distances are similar, the identity guarantee of RPA is not doing the work claimed; alternatively, measure reconstruction error of x_t^0 before and after one full projection step over a grid of refinement strengths to locate where the exact-equality property breaks.","tokens_in":12270,"feed_emoji":"🎥","tokens_out":4605,"duration_ms":45520,"temperature":0.7,"pith_summary":"AnyPortal aims to make video background replacement work out of the box, with no fine-tuning, by combining two kinds of pre-trained diffusion models: an image model that understands how light should fall on a subject in a new scene, and a video model that knows how frames should move together. The paper claims the hard part is not generating the new background but keeping the foreground identical to the input video. To solve this it introduces a Refinement Projection Algorithm that edits foreground details in pixel space and maps them back into the compact video latent space in a way that, by construction, leaves unedited regions unchanged. If the claim holds, users could 'teleport' a person or object into a new scene with stable lighting, stable geometry, and consistent identity, on a single consumer GPU.","feed_headline":"Zero-shot pipeline swaps video backgrounds without retraining","feed_subtitle":"AnyPortal combines image relighting with video diffusion priors to keep foreground pixels stable across frames.","key_machinery":"The Refinement Projection Algorithm (RPA) is the load-bearing mechanism. At each denoising step it takes the noise-free latent prediction x_t^0, decodes it to pixels, separates low- and high-frequency content, injects the input video's high-frequency foreground details, then re-encodes and projects the result back into latent space using the deterministic noise vector epsilon_hat = (x_t^0 - mu)/sigma. The identity property — if no refinement is applied, the projected latent equals the original latent — is what lets the pipeline refine foreground pixels without blurring or distorting the generated background. The two-step IC-Light harmonization with cross-frame attention supplies the relighti","core_discovery":"On the paper's own terms, the central discovery is that foreground-consistent video background replacement does not require training a video relighting model. AnyPortal demonstrates a modular, zero-shot pipeline in which the image diffusion model IC-Light performs background generation and relighting, and a video diffusion model supplies temporal consistency; the new ingredient is the Refinement Projection Algorithm, which guarantees pixel-level foreground preservation. The algorithm computes a deterministic noise vector from the current unrefined latent, then re-encodes a refined pixel-space video and reuses that same noise vector to sample the new latent. When no refinement is applied, thi","pith_inferences":["The identity property of RPA is really a statement about the VAE's reconstruction, so the same projection trick could plausibly be reused for other pixel-level video edits, such as recolorization, inpainting, or facial attribute editing, whenever the 3D latent space is too compact for direct latent manipulation.","A stress test the paper does not run: increasing refinement strength should eventually break the assumption that the deterministic noise vector stays valid; finding that threshold would map the method's operating range.","Because RPA only transfers high-frequency foreground detail, the method implicitly depends on the input video having sufficient resolution and sharp edges, a dependency the paper itself flags as a limitation."],"forward_implications":["Background replacement becomes an inference-only operation: any improvement in pre-trained image or video diffusion models can be dropped into the modular pipeline without retraining.","Foreground identity is preserved at pixel level in the edited region while background regions remain almost identical to the unrefined generation, addressing a failure mode common in text-guided video editing.","Both text-prompted and image-prompted backgrounds are supported with the same pipeline by only changing how the first background frame is produced.","The method runs on a single 24 GB GPU and produces a 49-frame 480x720 output in about 12 minutes, making the workflow practical outside professional studios.","Quantitative and user evaluations on 30 videos report the best temporal consistency, motion preservation, and foreground detail preservation among the zero-shot baselines compared."],"supporting_citations":[{"why":"Supplies the image relighting and background replacement model used in background generation and light harmonization.","marker":"[41]"},{"why":"Supplies the video diffusion prior and the compact 3D latent space in which generation and RPA operate.","marker":"[37]"},{"why":"Supplies the noise-and-denoise recipe used for light harmonization and temporal consistency enhancement.","marker":"[18]"},{"why":"Supplies the deterministic DDIM denoising equations that RPA modifies with its projection.","marker":"[30]"},{"why":"Supplies motion-aware background video generation from the first frame and tracked motion.","marker":"[9]"},{"why":"Removes foreground objects from generated backgrounds and inpaints videos during refinement.","marker":"[46]"},{"why":"Produces the foreground segmentation masks used throughout the pipeline.","marker":"[43]"},{"why":"Supplies the cross-frame attention mechanism used to keep style consistent across frames.","marker":"[35]"}],"fun_headline_variants":["Zero-shot video background swap without retraining","Training-free video background replacement via diffusion priors","AnyPortal: stable foregrounds in zero-shot video background swaps","Pixel-perfect foreground preservation in video background exchange","Swap video backgrounds zero-shot with consistent foregrounds"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The projection assumes the deterministic noise vector computed from the unrefined latent is still a valid reparameterization after the foreground refinement; if the refinement strongly shifts the VAE's mean or variance in unedited regions, that same noise vector can inject large deviations and degrade the background.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot video background swap without retraining","Training-free video background replacement via diffusion priors","AnyPortal: stable foregrounds in zero-shot video background swaps","Pixel-perfect foreground preservation in video background exchange","Swap video backgrounds zero-shot with consistent foregrounds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1114,"prompt_tokens":651,"completion_tokens":463,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":395,"completion_tokens_details":{"reasoning_tokens":390}},"tokens_in":395,"tokens_out":463,"duration_ms":5499,"temperature":1.0,"reasoning_tokens":390,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T22:08:28.435311+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a video with fast motion or a low-resolution foreground, run AnyPortal with RPA active and with RPA replaced by naive decode-refine-encode, and compare the background regions pixel-by-pixel against the no-refinement output. If the background distances are similar, the identity guarantee of RPA is not doing the work claimed; alternatively, measure reconstruction error of x_t^0 before and after one full projection step over a grid of refinement strengths to locate where the exact-equality property breaks.","supporting_citations":[{"cited_title":"Scaling in-the-wild training for diffusion-based illumination harmo- nization and editing by imposing consistent light transport","cited_arxiv_id":null,"evidence_quote":"Supplies the image relighting and background replacement model used in background generation and light harmonization."},{"cited_title":"Propainter: Improving propagation and transformer for video inpainting","cited_arxiv_id":null,"evidence_quote":"Removes foreground objects from generated backgrounds and inpaints videos during refinement."},{"cited_title":"Rerender a video: Zero-shot text-guided video-to-video translation","cited_arxiv_id":null,"evidence_quote":"Supplies the cross-frame attention mechanism used to keep style consistent across frames."}],"review_version":1}