FlexiAct: Towards Flexible Action Control in Heterogeneous Scenarios

Junhao Zhuang; Shiyi Zhang; Yansong Tang; Ying Shan; Zhaoyang Zhang

REVIEW 4 major objections 5 minor 2 cited by

FlexiAct: Towards Flexible Action Control in Heterogeneous Scenarios

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FlexiAct transfers actions from a reference video to an arbitrary target image without requiring alignment of layout, shape, or viewpoint, preserving both motion and appearance.

desk verdict A practical heterogeneous action-transfer method with a genuine RefAdapter trick, but the FAE frequency mechanism is under-evidenced; worth reviewing, not yet fully convincing. read the letter →

arxiv 2505.03730 v1 pith:7KKQJJSL submitted 2025-05-06 cs.CV cs.AIcs.MM

Shiyi Zhang , Junhao Zhuang , Zhaoyang Zhang , Ying Shan , Yansong Tang This is my paper · ORCID

classification cs.CVcs.AIcs.MM

keywords actiontransfervideocustomizationimage-to-videodiffusionfrequency-awareextractionattentionreweightingheterogeneousscenariosRefAdapterCogX-I2V

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes FlexiAct, a system that takes a reference video and an arbitrary target image and generates a new video in which the target subject performs the reference action, even when the two subjects differ in pose, skeleton, layout, viewpoint, or even species and artistic style. The authors argue this is the first action-transfer framework that does not need pose maps, depth, or matched first frames, and that it does so on a standard image-to-video diffusion backbone with two lightweight additions. If correct, FlexiAct would lower the cost of animation and video customization by replacing motion capture or hand animation with a single reference clip and one image.

What carries the argument

The mechanism is a timestep-dependent attention bias $W_{\text{bias}}$ (Eq. 1) applied to the attention weights between video tokens and a learnable frequency-aware embedding during denoising. The embedding is trained on a single reference video with random crops to prevent layout memorization; at inference, the bias is $\alpha=1$ for $t \geq 800$, a cosine ramp between $t_h=700$ and $t_l=800$, and zero below $700$, on the assumption that early denoising steps carry low-frequency motion and later steps carry high-frequency appearance. RefAdapter supplies the spatial flexibility by randomly sampling the conditioning frame from the full video during training and replacing the first temporal embedding with the conditioned image, breaking the strict first-frame constraint of I2V models.

What would settle it

Run the FAE inference with the bias schedule inverted—high bias in late steps, low in early steps—and measure motion fidelity and appearance consistency on the paper's 250-pair benchmark; if motion fidelity does not drop substantially and appearance does not leak, the timestep-frequency story is not what is carrying the result.

Watch

Extended reading notes

Core claim

The central claim is that action transfer in heterogeneous scenarios reduces to two problems—spatial structure adaptation and action extraction—and that FlexiAct solves both: RefAdapter conditions generation on an arbitrary frame of the reference video with a small LoRA, enabling the model to adapt motion to new layouts and shapes, and FAE trains per-video frequency-aware embeddings whose attention to video tokens is reweighted by timestep during inference, amplifying motion in early denoising steps. The paper reports that this pipeline outperforms a MotionDirector baseline on CogVideoX-I2V and its own ablations on motion fidelity and appearance consistency, with human raters preferring the outputs.

Load-bearing premise

The argument depends on the claim that the trained frequency-aware embedding's attention actually separates motion (low frequency) from appearance (high frequency) in the way the timestep schedule assumes, so that adding bias during early steps transfers motion without leaking the reference video's appearance; that claim rests on visual inspection of attention maps and ablations, not on a quantitative frequency analysis.

Editorial extensions

If this is right

Pose-based action customization becomes unnecessary for many tasks: users can transfer an action from a single clip to any image of a human, animal, or drawn character, without extracting skeletons or depth.
Adding the same attention-bias mechanism to other image-to-video diffusion models could give them heterogeneous action transfer without retraining the base model.
Because FAE requires per-reference-video optimization (1,500 to 3,000 training steps), the method is not feed-forward; a single reference clip costs time to embed, which the authors identify as a limitation.
The 250-pair benchmark with 25 action categories and cross-domain targets gives the community a common test bed, but the paper's reported numbers depend on CLIP-based metrics and preference votes, not on direct action recognition.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: if the frequency separation is real and robust, the same $W_{\text{bias}}$ trick could be extended to video editing beyond action transfer—e.g., transferring camera motion or object trajectories from one clip to another, since these are also low-frequency components.
Editorial inference: the hand-set hyperparameters ($t_h=700$, $t_l=800$, $\alpha=1$) are likely backbone-specific; applying FAE to a different diffusion model would require re-measuring where the motion-to-appearance transition occurs, suggesting a per-model calibration step.
Editorial inference: a cleaner test of the frequency story would be to compute the spectral content of the attention maps directly (e.g., via a spatial or temporal Fourier transform) and correlate it with timestep; the paper's visual inspection of attention maps supports but does not prove the claim.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Terry, quick take on FlexiAct: it's a genuinely useful engineering contribution to action transfer, with a nice trick for loosening I2V's first-frame constraint, but the mechanistic claim about frequency-aware embeddings is supported only by eyeballed attention maps. The paper deserves a serious referee, but the next revision should tighten the evidence.

What's new: unlike pose-guided or global-motion methods, FlexiAct transfers an action from a reference video to an arbitrary target image without requiring matching skeletons, layouts, or viewpoints. The two components are RefAdapter, a LoRA-based image conditioner trained with a randomly selected frame (not the first frame) as the condition and with the first latent embedding replaced by the condition embedding, and FAE, a per-video learnable embedding plus an inference-time timestep-dependent bias on attention weights (Eq. 1) that boosts early denoising steps to emphasize motion. The random-frame conditioning is a simple, sensible departure from standard I2V training and likely generalizes. The qualitative results, especially cross-domain transfers (human to animal, real to animated), look convincing, and the authors release code and weights.

Soft spots: the frequency interpretation is the weakest link. Figure 2 shows attention maps at three timesteps, but there is no quantitative frequency decomposition, no per-layer/head statistics, and no causal test that the bias actually amplifies low-frequency motion. More importantly, FAE is trained without RefAdapter loaded (Section 3.4), yet at inference RefAdapter is loaded; the attention maps may not describe the deployed configuration. The schedule (alpha=1, t_h=700, t_l=800) is hand-set and justified by artifact analysis in Figure 11 on the same examples, so there is some circularity. Evaluation is also thinner than I'd like: 250 self-constructed pairs, no released benchmark, no error bars on the automatic metrics, and a 5-rater human study. The baseline set is just one reimplemented global-motion method and a base model; recent methods like MotionClone or Motion Inversion are not compared. There's also a citation slip in Section 4.1 where MotionDirector is attributed to Tu et al. 2024a instead of Zhao et al. 2023.

None of this kills the central capability claim—the ablations show both components matter, and the qualitative evidence is strong. But the 'frequency-aware' framing is more of an interpretation than a demonstrated mechanism. For a paper at this venue, that's acceptable if the engineering works; for a journal version, I'd want the frequency claim quantified.

Bottom line: send it to review. The ideas are clear, the method is reproducible (code released), and the results are interesting to AIGC and video-generation folks. With a bit more evidence on the mechanism and a stronger evaluation, it could be a solid reference.

Referee Report

4 major / 5 minor

Summary. The paper proposes FlexiAct, an action-transfer framework built on CogVideoX-I2V that takes a reference video and an arbitrary target image and generates a video of the target subject performing the reference action, without requiring alignment of layout, skeleton, or viewpoint. The method has two novel components: RefAdapter, a lightweight LoRA-based image-conditioned adapter trained with randomly sampled condition frames to support spatial adaptation and appearance consistency, and FAE (Frequency-aware Action Extraction), which trains per-video learnable embeddings and, at inference, adds a hand-crafted attention bias W_bias (Eq. 1) at early denoising timesteps based on qualitative observations of attention maps. The paper evaluates on a self-constructed benchmark of 250 reference-video/target-image pairs with 25 action categories, comparing against a reimplemented MD-I2V baseline and a BaseModel, using automatic metrics and a human preference study. The authors claim state-of-the-art flexible action transfer, the first denoising process-based action extraction framework, and release code and model weights.

Significance. If the central claim holds, FlexiAct would be a practically useful contribution to action customization in heterogeneous scenarios, since it relaxes the spatial-alignment constraints of pose-based and global-motion methods while maintaining identity. The paper has clear strengths: it includes separate ablations for both proposed components, uses human preference evaluation, reports qualitative results across diverse subjects including animals and cross-domain characters, and promises release of code and model weights. However, the central 'frequency-aware' mechanism is supported only by qualitative attention-map inspection, the automatic metrics lack error bars and significance tests, and the human study is small; these issues make the significance conditional on additional evidence rather than established by the current manuscript.

major comments (4)

[Section 4.2, Table 1] The automatic metrics are reported as point estimates with no error bars, confidence intervals, or significance tests. For example, the Motion Fidelity gap between w/o FAE (0.3614) and Ours (0.4103), and the Appearance Consistency gap between w/o RefAdapter (0.9021) and Ours (0.9162), could be within run-to-run or seed-to-seed variation. Please report variance over seeds or paired per-item statistics with a significance test. The evaluation set is also self-constructed and not released, which limits reproducibility of these numbers.
[Section 3.4, Eq. (1), Figure 2] The load-bearing premise that the frequency-aware embeddings separate low-frequency motion from high-frequency appearance is supported only by a qualitative attention-map montage. There is no quantitative frequency decomposition, no per-layer/head/timestep statistics, and no sensitivity analysis for the parameters alpha, t_h, and t_l. Moreover, FAE is trained without RefAdapter (Section 3.5) but deployed with it, so the Figure 2 attention maps characterize the training configuration rather than the deployed configuration; if RefAdapter's LoRA shifts the attention distribution, the fixed bias may not behave as claimed. I recommend adding a quantitative frequency analysis and a control experiment that varies the schedule direction (e.g., applying the bias at late timesteps or at random timesteps) to show that the specific early-timestep schedule, and not any positive bias, is what drives the gain.
[Section 4.2, Human Evaluation] The human study uses 5 raters, each comparing 50 randomly selected pairs, and the paper reports preference percentages without confidence intervals, significance tests, or inter-rater agreement. Additionally, the statement that '5 raters assessed each generated video' is inconsistent with the described protocol in which each rater compares 50 random pairs. Please clarify the protocol and provide statistical support for the preference results, e.g., bootstrap confidence intervals or a paired significance test, along with details of rater instructions.
[Section 4.1, Comparison Methods] The baseline MD-I2V is a reimplementation of MotionDirector on CogVideoX-I2V, but no evidence is provided that the reimplementation matches MotionDirector's original behavior or that hyperparameters were tuned for the I2V setting. The text says 'identical training settings to our methods' while also separately reporting MotionDirector's original 3,000/300 LoRA steps, which is confusing. To substantiate the claim of surpassing existing methods, please include at least one additional global-motion baseline and, on the human subset where pose is available, a pose-based baseline.

minor comments (5)

[Eq. (1)] The cosine transition function uses the variable x in its argument but t elsewhere; please use t consistently for the denoising timestep.
[Table 1] The table header is ambiguous: 'Appearance Consistency↑' appears before the human-evaluation columns 'Motion Consistency Appearance Consistency', making it unclear which columns belong to automatic versus human evaluation. Also, 'are prefer over' should be 'are preferred over'.
[Section 4.1] The citation 'MotionDirector [Tu et al. 2024a]' is inconsistent with the later 'MD-I2V [Zhao et al. 2023]'; in the reference list, Tu et al. 2024a is MotionEditor. Please correct the citation.
[Figure 3] The notation 'N Gaussian noise' is unclear; please specify that N is the noise latent and state its shape or refer explicitly to the latent noise described in Section 3.2.
[Section 5] The limitation that FAE requires per-video optimization is acknowledged, but please add a brief discussion of computational cost in practice, since this affects the practical utility of the method.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are grounded in separate training stages, external benchmarks, and ablations.

full rationale

I examined the derivation chain of FlexiAct. The central capability claim (action transfer from a reference video to an arbitrary target image without layout/shape/viewpoint alignment) is implemented by two separately trained components: RefAdapter, trained once on Miradata with standard denoising objectives, and FAE, whose per-video frequency-aware embedding is trained on the reference video itself. The inference-time reweighting W_bias in Eq. (1) is a hand-set schedule (alpha=1, t_h=700, t_l=800), not a parameter fitted to the evaluation metric, and its contribution is tested by ablations (w/o FAE vs. Ours) on a 250-pair benchmark with human evaluation. The only overlapping-author citation is Miradata, which is used as training data rather than as a load-bearing justification for correctness. No equation in the paper reduces to another by construction, and no fitted quantity is renamed as a prediction. The qualitative attention-map evidence in Figure 2 is offered as motivation rather than as the sole proof; even if that evidence is weak, weakness of evidence is not circularity. Therefore the paper does not exhibit self-definitional, fitted-input, or self-citation-circular reasoning.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The central claim rests on two learned components: a LoRA-based RefAdapter conditioned on random frames, and a per-video frequency-aware embedding. The free parameters are the inference-time attention schedule (alpha, th, tl) and per-video training steps. The main domain assumptions are the capability of the pretrained base model, the interpretation of attention maps, and the validity of the evaluation metrics.

free parameters (3)

alpha (attention bias strength) = 1
Sets the magnitude of W_bias added to attention during FAE inference (Eq. 1); chosen by hand based on qualitative comparisons, no sweep reported.
t_l and t_h (low/high frequency timesteps) = t_l=800, t_h=700
Boundaries of the bias transition in Eq. 1; selected via failure analysis in Figure 11 where hard thresholds at 700 or 800 cause appearance leakage or motion errors.
FAE training steps per reference video = 1500-3000
Number of fine-tuning steps for the per-video embedding; chosen according to action complexity, affects the captured motion/appearance tradeoff.

assumptions (4)

domain assumption CogVideoX-I2V provides a capable base image-to-video model whose latent space supports identity-preserving synthesis.
The entire pipeline is built on this pretrained model (Section 3.2); no independent comparison against other I2V backbones is given.
domain assumption The attention maps between frequency-aware embeddings and video tokens reflect motion (low-frequency) and appearance (high-frequency) content.
The central observation in Section 3.4 and Figure 2; the paper does not measure frequency spectra or causally verify that attention corresponds to motion.
domain assumption Miradata (Ju et al., self-cited) is a suitable training corpus for learning flexible image conditioning.
RefAdapter is trained on 42,000 videos from this dataset (Section 3.3); dataset quality is asserted via self-citation.
domain assumption The CLIP-based and CoTracker-based metrics measure appearance and motion fidelity adequately for heterogeneous action transfer.
Evaluation in Section 4.2 relies on these proxies; no correlation with human judgments is reported.

invented entities (1)

Frequency-aware Embedding
purpose: Learned tokens concatenated to MMDiT inputs, trained per reference video to capture motion and appearance, then reweighted at inference to extract action.
The embedding is validated only through the paper's own ablations and qualitative examples; no external dataset or formal analysis confirms that it separates frequency bands.

how reviews work

0 comments

Cite this review

Pith. "Pith review of FlexiAct: Towards Flexible Action Control in Heterogeneous Scenarios." pith.science (2026). https://pith.science/paper/7KKQJJSL

@misc{pith2026250503730,
  author       = {Pith},
  title        = {Pith review of: FlexiAct: Towards Flexible Action Control in Heterogeneous Scenarios},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7KKQJJSL}},
  note         = {Machine review of arXiv:2505.03730}
}

read the original abstract

Action customization involves generating videos where the subject performs actions dictated by input control signals. Current methods use pose-guided or global motion customization but are limited by strict constraints on spatial structure, such as layout, skeleton, and viewpoint consistency, reducing adaptability across diverse subjects and scenarios. To overcome these limitations, we propose FlexiAct, which transfers actions from a reference video to an arbitrary target image. Unlike existing methods, FlexiAct allows for variations in layout, viewpoint, and skeletal structure between the subject of the reference video and the target image, while maintaining identity consistency. Achieving this requires precise action control, spatial structure adaptation, and consistency preservation. To this end, we introduce RefAdapter, a lightweight image-conditioned adapter that excels in spatial adaptation and consistency preservation, surpassing existing methods in balancing appearance consistency and structural flexibility. Additionally, based on our observations, the denoising process exhibits varying levels of attention to motion (low frequency) and appearance details (high frequency) at different timesteps. So we propose FAE (Frequency-aware Action Extraction), which, unlike existing methods that rely on separate spatial-temporal architectures, directly achieves action extraction during the denoising process. Experiments demonstrate that our method effectively transfers actions to subjects with diverse layouts, skeletons, and viewpoints. We release our code and model weights to support further research at https://shiyi-zh0408.github.io/projectpages/FlexiAct/

Figures

Figures reproduced from arXiv: 2505.03730 by the authors.

**Figure 1.** Visualization for our FlexiAct results. Given a target image, FlexiAct transfers actions from a reference video to the target subject, achieving accurate motion adaptation and appearance consistency even in heterogeneous scenarios with varying spatial structures or cross-domain subjects. Action customization involves generating videos where the subject performs actions dictated by input control signals. Current meth… view at source ↗

**Figure 2.** Attention maps between our frequency-aware embeddings and video tokens in the MMDiT at different denoising timesteps. Our embeddings focus on low-frequency motion information (e.g., motion regions) in early denoising stages and shift to high-frequency details in later stages. 2.2 Predefined signal-based Action Customization Action customization methods based on predefined signals, such as pose, depth, and edges, tra… view at source ↗

**Figure 3.** Overview of FlexiAct. (1) The upper part illustrates RefAdapter’s training, which conditions arbitrary frames to enable transitions across diverse spatial structures. (2) The lower part outlines FAE’s training and inference, where attention weights of video tokens to the frequency-aware embedding are dynamically adjusted based on timesteps, facilitating action extraction. 3.2 Basis Image-to-Video Diffusion Model We … view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: Results of transferring “turning” action to the target image using the [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Qualitative comparison of action transfer from reference video (Ref Video) to target images with varying spatial structures. Red boxes highlight regions [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: Qualitative results of ablation study.We ablate Frequency-aware Action Extraction (FAE) and RefAdapter, comparing the action transfer results from reference videos (Ref Video) to different subjects. Ablating FAE reduces action accuracy, demonstrating its effectiveness …

**Figure 7.** Figure 7: FlexiAct can transfer actions to diverse subjects while maintaining both appearance consistency with the target subject and action consistency with [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

**Figure 8.** Figure 8: Examples of human action transfer using FlexiAct. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

**Figure 9.** Figure 9: Examples of action transfer between animals using FlexiAct. [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]

**Figure 10.** Figure 10: Examples of action transfer from humans to animals using FlexiAct. [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

MIND-V: Hierarchical World Model for Long-Horizon Robotic Manipulation with RL-based Physical Alignment
cs.RO 2025-12 conditional novelty 6.0 of 10

MIND-V generates long-horizon robot manipulation videos by decomposing instructions into sub-tasks with a VLM, encoding plans into a structured bridge, and fine-tuning a video diffusion model with a V-JEPA2-based phys...
O-DisCo-Edit: Object Distortion Control for Unified Realistic Video Editing
cs.CV 2025-09 conditional novelty 6.0 of 10

A video editor trained on randomly distorted objects, then steered by adaptive noise at inference, is claimed to surpass dedicated and unified editors across eight tasks with far less training.

Reference graph

Works this paper leans on

3 extracted references · 1 canonical work pages · cited by 2 Pith papers

[2019]

In NeurIPS

First Order Motion Model for Image Animation. In NeurIPS. Aliaksandr Siarohin, Oliver J Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov

work page
[2021]

Motion representations for articulated animation. In CVPR. Shuyuan Tu, Qi Dai, Zhi-Qi Cheng, Han Hu, Xintong Han, Zuxuan Wu, and Yu-Gang Jiang. 2024a. Motioneditor: Editing video motion via content-aware diffusion. In CVPR. Shuyuan Tu, Zhen Xing, Xintong Han, Zhi-Qi Cheng, Qi Dai, Chong Luo, and Zux- uan Wu. 2024b. StableAnimator: High-Quality Identity-Pr...

arXiv 2024
[2024]

arXiv preprint arXiv:2408.06070 (2024)

ControlNeXt: Powerful and Efficient Control for Image and Video Generation. arXiv preprint arXiv:2408.06070 (2024). Haonan Qiu, Menghan Xia, Yong Zhang, Yingqing He, Xintao Wang, Ying Shan, and Ziwei Liu. 2023. Freenoise: Tuning-free longer video diffusion via noise rescheduling. arXiv preprint arXiv:2310.15169 (2023). Alec Radford, Jong Wook Kim, Chris H...

arXiv 2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [2019]

In NeurIPS

First Order Motion Model for Image Animation. In NeurIPS. Aliaksandr Siarohin, Oliver J Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov

work page

[2] [2021]

Motion representations for articulated animation. In CVPR. Shuyuan Tu, Qi Dai, Zhi-Qi Cheng, Han Hu, Xintong Han, Zuxuan Wu, and Yu-Gang Jiang. 2024a. Motioneditor: Editing video motion via content-aware diffusion. In CVPR. Shuyuan Tu, Zhen Xing, Xintong Han, Zhi-Qi Cheng, Qi Dai, Chong Luo, and Zux- uan Wu. 2024b. StableAnimator: High-Quality Identity-Pr...

arXiv 2024

[3] [2024]

arXiv preprint arXiv:2408.06070 (2024)

ControlNeXt: Powerful and Efficient Control for Image and Video Generation. arXiv preprint arXiv:2408.06070 (2024). Haonan Qiu, Menghan Xia, Yong Zhang, Yingqing He, Xintao Wang, Ying Shan, and Ziwei Liu. 2023. Freenoise: Tuning-free longer video diffusion via noise rescheduling. arXiv preprint arXiv:2310.15169 (2023). Alec Radford, Jong Wook Kim, Chris H...

arXiv 2024