Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Direct Motion Models for Assessing Generated Videos

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that evaluating generated videos by auto-encoding point trajectories — rather than comparing pixels or action-recognition features — yields a motion metric that is markedly more sensitive to temporal distortions and…

desk verdict TRAJAN is a genuinely useful new motion-quality metric for generated videos; the distribution-level and human-correlation evidence holds up, but the single-video mode rests on a premise the paper only partially validates. read the letter →

arxiv 2505.00209 v1 pith:N3SI7W4C submitted 2025-04-30 cs.CV cs.LG

Kelsey Allen , Carl Doersch , Guangyao Zhou , Mohammed Suhail , Danny Driess , Ignacio Rocco , Yulia Rubanova , Thomas Kipf
show 4 more authors
Mehdi S. M. Sajjadi Kevin Murphy Joao Carreira Sjoerd van Steenkiste
This is my paper · ORCID
classification cs.CVcs.LG
keywords pointtracksvideogenerationevaluationtrajectoryautoencodertemporalconsistencyFréchetdistanceAverageJaccardhumanofqualitygenerativemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to fix a blind spot in how generated videos are judged: models can produce frames that look fine but move wrongly, and standard metrics such as FVD miss this because they lean on appearance and action-recognition features. The authors propose TRAJAN, a trajectory autoencoder that compresses a video's point tracks into a fixed latent vector and reconstructs held-out tracks from that vector. Because the model is trained only on real-world motion, its reconstruction error—measured by Average Jaccard—acts as a per-video score for motion plausibility, while Fréchet distance over the latents compares whole datasets and L2 distance compares video pairs. On synthetic temporal-distortion tests and on human ratings for videos from open-source generators, TRAJAN outperforms I3D, VideoMAE, motion histograms, optical-flow warping, and next-frame prediction baselines. If the paper's central claim holds, video quality evaluation need not depend on a reference dataset or on expensive human studies.

What carries the argument

The central object is the TRAJAN trajectory autoencoder operating on point tracks. Point tracks are temporally extended (x, y, occlusion) trajectories of arbitrary points, extracted by BootsTAPIR; they separate motion from appearance because they never reconstruct pixels. TRAJAN embeds each track with sinusoidal position encodings, summarizes it with self-attention using occlusion flags as masks, and compresses the whole set with a Perceiver-style cross-attention into 128×64 latent tokens. A decoder consumes a query point (xq, yq, tq) and predicts the full track through that point; training on held-out queries forces the latent to capture a dense motion field. Average Jaccard, borrowed from TAP-Vid, measures reconstruction accuracy across pixel thresholds and serves as the per-video quality score.

What would settle it

Compare TRAJAN's Average Jaccard on a set of real videos containing unusual but physically realistic motion (e.g., cloth flapping, water splashes, or slow-motion impacts) against ordinary action videos: if the unusual-but-real videos systematically receive low reconstruction scores while humans judge them realistic, the per-video ordering claim collapses. Alternatively, take a single real video and re-render it with the same motion but different textures; if Average Jaccard changes substantially, the metric is not purely motion-based.

Watch

Extended reading notes

Core claim

The central claim is that point tracks are a better substrate than pixels or action-recognition features for measuring motion quality in generated video, and that an autoencoder trained on those tracks can serve as a single unified metric across all three evaluation settings. TRAJAN encodes a variable-sized, orderless set of point trajectories—extracted by BootsTAPIR—into a fixed 128×64 latent using a Perceiver-style transformer, and is trained to reconstruct query tracks that were held out from the input, making the representation invariant to which points were sampled and to occlusion. The same model then yields a distribution metric (Fréchet distance between latent Gaussian fits), a pairwise metric (L2 distance between latents), and an individual-video metric (Average Jaccard reconstruction error). The paper reports that the Fréchet variant is more than three times as sensitive to synthetic temporal distortions as prior backbones, that the latent distance ranks WALT-generated videos near their real counterparts when motion is correct even if pixels diverge, and that the reconstruction score correlates with human ratings of consistency, realism, and interactions better than all tested alternatives on EvalCrafter and VideoPhy data, while remaining competitive on optical-flow warping.

Load-bearing premise

The load-bearing assumption is that an autoencoder trained on ordinary real-world point tracks will find realistic motion easier to reconstruct than unrealistic motion, so its reconstruction error ranks videos by perceptual quality — and the paper validates this only indirectly through correlations with human ratings, not by independently confirming that unusual-but-real motions are reconstructed well.

Editorial extensions

If this is right

  • On the UCF-101 elastic-corruption test, the TRAJAN Fréchet distance is markedly more sensitive to temporal distortions than FVD backbones, and the per-video Average Jaccard nearly matches the distribution-level sensitivity, suggesting per-video scores can substitute for reference-based comparisons when no reference is available.
  • A reference-free per-video motion score means one can rank individual generated videos, monitor training checkpoints, and filter outputs without needing the generator's training distribution, which FVD requires.
  • TRAJAN's latent distance ranks generated WALT videos that share motion with the real continuation as closer than appearance-based latent distances do, even when pixel error is large, indicating a motion-only similarity signal.
  • The reconstruction error correlates with human ratings across multiple dimensions besides motion—appearance consistency, realism, and object interactions—and tracks which generator humans prefer overall, suggesting motion quality carries much of the perceived-quality signal.
  • Because Average Jaccard can be computed per point and per frame, TRAJAN localizes where and when generative inconsistencies occur, as shown for morphing hands and gloves.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If realistic motion is indeed easier to reconstruct than unrealistic motion, the same autoencoder could be repurposed as a differentiable reward or loss for training or fine-tuning generative video models, not just an evaluation metric.
  • The finding that human raters disagree substantially with one another suggests per-video quality is partly subjective; a practical direction is to build a metric that predicts a distribution of human ratings rather than a single scalar, or to calibrate any metric against rater agreement.
  • The paper's motion-only distance could be combined with prompt-semantic metrics such as CLIPScore to yield a two-axis evaluation: adherence to the text and physical plausibility of the motion.
  • One testable extension is to apply TRAJAN to unusual but physically realistic motion—fluids, cloth, or slow motion—where reconstruction difficulty may not track perceived realism; if it fails there, the ordering claim is narrower than human perception.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes TRAJAN, a trajectory autoencoder trained on point tracks extracted by BootsTAPIR, and uses its latent space or reconstruction error (Average Jaccard) as a video quality metric. The authors claim three modes of use: distribution-level comparison via Fréchet distance, paired-video comparison via latent L2 distance, and single-video evaluation via reconstruction error. They report that TRAJAN is markedly more sensitive to synthetic temporal distortions on UCF-101 than appearance-based and histogram-based alternatives, and that its per-video scores correlate with human ratings of consistency, realism, interaction quality, and speed on EvalCrafter and VideoPhy, outperforming a range of baselines. They also introduce a new human-study protocol and a spatiotemporal error localization tool.

Significance. If the claims hold, TRAJAN would be a useful contribution to video generation evaluation, particularly for distribution-level sensitivity to temporal distortions and for the interpretability of localized motion inconsistencies. The synthetic UCF-101 experiment is a clean and convincing demonstration of motion sensitivity, and the human study is a substantial empirical contribution covering two public datasets and multiple rating dimensions. The promise of code release and the detailed experimental appendix are strengths. However, the central per-video claim rests on an untested mechanism, the paired-video mode lacks quantitative validation, and the human-correlation evidence is modest and possibly affected by model selection on human labels. These issues currently prevent accepting the paper's strongest claims at face value.

major comments (4)
  1. [Section 3.1 and Section 5.3] The per-video metric rests on the premise stated in Section 3.1 that, because TRAJAN was trained on real motions, unrealistic motions will be more difficult to reconstruct. This premise is never tested independently of the human-rating correlations or the synthetic UCF-101 experiment. The paper itself provides a counterexample in Figure 6, where a physically implausible glass collapse is scored positively by all metrics. Moreover, because Average Jaccard is computed against BootsTAPIR tracks, failures of the tracker on generated videos with morphing or disappearing content will inflate reconstruction error regardless of whether the underlying motion is unrealistic, confounding the interpretation of the metric. I request either a direct test of the mechanism (e.g., perturbing motion while controlling trackability) or a careful narrowing of the claim to what the evidence supports, namely that AJ captures smoothness or trackability rather than physical realism.
  2. [Appendix B.2.1] The description of TRAJAN training discloses that the Huber and occlusion loss weights were selected in initial experiments by their correlation with human judgements of realism in generated videos. Since the headline result in Section 5.3 is that TRAJAN predicts human ratings better than alternatives, this creates a fairness problem: TRAJAN is tuned with respect to the evaluation target, while the comparison methods (VideoMAE, I3D, RAFT, MooG) are used as off-the-shelf. The authors should either report TRAJAN with default or equally weighted losses, or quantify how much of the advantage in Table 2 depends on this human-label-based selection.
  3. [Section 5.2 and Table 1] The paired-video mode is validated only by showing low correlation with PSNR and SSIM and by qualitative examples. Low correlation with pixel-based metrics establishes that TRAJAN measures something different from pixel error, but it does not establish that the latent distance correctly captures similarity in motion. Without a controlled experiment or human judgments for paired comparisons, the abstract's claim that TRAJAN can compare one generated video to one ground truth video is not supported. Please add a quantitative validation of the paired-video mode, for example by using synthetic motion-preserving or motion-degrading perturbations and measuring whether the ranking of pairs matches ground truth.
  4. [Tables 2 and 4] The human-rating correlations are modest (Spearman coefficients of approximately 0.09 to 0.32 in Table 2, and 0.24 to 0.33 in the motion-controlled analysis of Table 4), yet the conclusion states that reconstruction error 'correlates remarkably well' with human judgments. No confidence intervals or significance tests are reported, and the inter-rater standard deviations are high (around 0.5). For several cells the advantage over baselines is small or tied (e.g., VideoPhy Interacts: TRAJAN 0.09 versus I3D 0.09). The authors should report confidence intervals, compare effect sizes with rater noise, and soften the conclusion accordingly.
minor comments (5)
  1. [Section 1] There is a typo in the first bullet list: 'synthethic' should be 'synthetic'.
  2. [Abstract] The phrase 'as few as one generated and one ground truth' in the distribution-level comparison setting is confusing, since a Fréchet distance between two one-sample distributions is just the L2 distance between two embeddings. Please clarify what is meant.
  3. [Appendix B.2.1] The sentence 'with average points within threshold of 85.3' should presumably read '85.3%' to indicate a percentage of points within the threshold.
  4. [Figure 4] The left panel of Figure 4 would benefit from error bars or a description of how the 'average temporal sensitivity' is computed across corruption levels, since the main distribution-level claim depends on this quantity.
  5. [Section 5.2] In Table 1, the interpretation that 'distances in latent space are not well correlated with PSNR or SSIM' would be clearer if the authors stated explicitly that this is evidence of orthogonality rather than of correctness.

Circularity Check

1 steps flagged · score 2.0 of 10

No structural circularity; minor coupling from loss-weight selection to the human-judgment evaluation objective.

  1. fitted input called prediction [Appendix B.2.1 (TRAJAN training details)]
    "In initial experiments, we found that setting these weights equally led to worse performance in correlating with human judgements of realism in generated videos."

    The headline evaluation is the Spearman correlation between TRAJAN's per-video score and human ratings of realism/consistency on generated videos. This sentence states that the two scalar loss weights (Huber vs occlusion) were selected by exactly that criterion. Unless the 'initial experiments' used a disjoint held-out set of videos and raters (not stated in the paper), the reported correlations are partly a selection result: the model was tuned to correlate with human realism judgments and then shown to correlate with human realism judgments.

full rationale

The central derivation is otherwise self-contained. TRAJAN is trained as an autoencoder on real point tracks from BootsTAPIR without any human labels, and its quality is assessed on (a) synthetic elastic distortions of UCF-101 from Ge et al. (2024) and (b) human ratings of videos from VideoPhy and EvalCrafter that were not used as training targets. The 'unrealistic motions are harder to reconstruct' premise in Section 3.1 is an assumption rather than a circular equation; the paper even supplies a counterexample in Figure 6 (the collapsing glass), where smooth but physically implausible motion is scored positively. That is a correctness/validity limitation, not a definitional circularity. The citations to BootsTAPIR (Doersch et al., 2024) and TAP-Vid (Doersch et al., 2022) are prior public tools/benchmarks by overlapping authors; they are used as inputs and evaluation utilities, not as unverified uniqueness claims. The only genuine coupling is the loss-weight selection in Appendix B.2.1, which uses the same human-judgment correlation objective as the headline per-video evaluation. This is a minor, non-central circularity, so the score is 2 rather than 0.

Assumptions & free parameters 1 free parameters · 5 assumptions · 1 invented entities

TRAJAN is an empirical, learned metric, so the main axioms are modeling assumptions rather than mathematical premises. The most important are: point tracks capture motion while suppressing appearance; an autoencoder trained on real tracks will rank realistic motions as easier to reconstruct; and the training video distribution is representative of real motion. The only hyperparameter that touches the evaluation objective is the loss weight, tuned with human ratings. No physical entities are invented; TRAJAN itself is a learned model with external validation.

free parameters (1)
  • TRAJAN loss weights (Huber and occlusion cross-entropy) = Huber weight 5000, occlusion cross-entropy weight 1e-8
    Chosen empirically by the authors after 'initial experiments' using correlation with human judgements of realism in generated videos (Appendix B.2.1). This couples a design choice to the evaluation objective.
assumptions (5)
  • domain assumption Point tracks separate motion from appearance sufficiently for quality assessment
    The method assumes that a set of 2D point trajectories captures motion without needing to reconstruct appearance, so that motion quality can be scored independently of texture (Section 1, Figure 1).
  • domain assumption An autoencoder trained on real point tracks will reconstruct realistic motions more accurately than unrealistic ones
    The per-video metric (Average Jaccard) is interpreted as a quality score under this inductive bias. The paper provides empirical support but no formal justification (Sections 3.1 and 5.3).
  • domain assumption The YouTube lifestyle and one-shot video distribution is representative of realistic motion in evaluation domains
    TRAJAN is trained on 15M filtered YouTube videos, while it is tested on UCF-101, Kinetics-600, EvalCrafter, and VideoPhy. Domain shift could bias the metric against uncommon but valid motions (Appendix B.2.1).
  • standard math Frechet distance assumes multivariate Gaussian latents
    Distribution-level comparison uses the closed-form Frechet distance following FVD; no normality test is reported for TRAJAN latents (Section 5.1).
  • domain assumption Human ratings are treated as the ground truth for video quality despite strong rater disagreement
    The paper validates all metrics against its own human study, but inter-rater standard deviation is about half of the within-rater variation, so human judgments are noisy (Tables 2 and 7).
invented entities (1)
  • TRAJAN trajectory autoencoder independent evidence
    purpose: Learned fixed-size latent representation of point tracks, used as a multi-mode video quality metric and for spatiotemporal error localization.
    The paper provides falsifiable predictions (reconstruction errors and distances) evaluated against synthetic distortions and human ratings on external datasets, and references a project page with code.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Direct Motion Models for Assessing Generated Videos." pith.science (2026). https://pith.science/paper/N3SI7W4C

@misc{pith2026250500209,
  author       = {Pith},
  title        = {Pith review of: Direct Motion Models for Assessing Generated Videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N3SI7W4C}},
  note         = {Machine review of arXiv:2505.00209}
}
read the original abstract

A current limitation of video generative video models is that they generate plausible looking frames, but poor motion -- an issue that is not well captured by FVD and other popular methods for evaluating generated videos. Here we go beyond FVD by developing a metric which better measures plausible object interactions and motion. Our novel approach is based on auto-encoding point tracks and yields motion features that can be used to not only compare distributions of videos (as few as one generated and one ground truth, or as many as two datasets), but also for evaluating motion of single videos. We show that using point tracks instead of pixel reconstruction or action recognition features results in a metric which is markedly more sensitive to temporal distortions in synthetic data, and can predict human evaluations of temporal consistency and realism in generated videos obtained from open-source models better than a wide range of alternatives. We also show that by using a point track representation, we can spatiotemporally localize generative video inconsistencies, providing extra interpretability of generated video errors relative to prior work. An overview of the results and link to the code can be found on the project page: http://trajan-paper.github.io.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Measuring 3D Spatial Geometric Consistency in Dynamic Video Generation

    cs.CV 2026-03 conditional novelty 6.0 of 10

    SGC quantifies 3D geometric consistency of generated videos by measuring divergence among local camera poses estimated only on static background sub-regions.

  2. Generative Action Tell-Tales: Assessing Human Motion in Synthesized Videos

    cs.CV 2025-12 conditional novelty 6.0 of 10

    A learned human-action manifold, combining 3D pose, 2D keypoints, appearance, and motion derivatives, scores generated videos by distance to real-action centroids and embedding smoothness, beating prior metrics on hum...

Reference graph

Works this paper leans on

13 extracted references · 11 canonical work pages · cited by 2 Pith papers

  1. [1]

    Gen2 (December) (Esser et al., 2023)

    work page 2023

  2. [2]

    MoonValley (MoonValley)

    work page

  3. [3]

    PikaLab (December) (Pika)

    work page

  4. [4]

    Show-1 (Zhang et al., 2024)

    work page 2024

  5. [5]

    VideoCrafter 1.0 (Chen et al., 2023)

    work page 2023

  6. [6]

    Hotshot-XL (Hotshot-XL)

    work page

  7. [8]

    Gen2 (Esser et al., 2023)

    work page 2023

  8. [9]

    Floor33 Pictures (Pictures)

    work page

Show all 13 references
  1. [10]

    ZeroScope (Sterling)

  2. [11]

    true positives

    ModelScope (Wang et al., 2023a) We obtain 1144 videos in this way, which we preprocess to256× 256 resolution. 23 Direct Motion Models for Assessing Generated Videos (a) Spatial-only corruptions - level 1.1 (b) Spatial-only corruptions - level 1.2 (c) Spatial-only corruptions -...

    2024

  3. [625]

    something something

    Springer-Verlag, Oct. 2012. J. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In IEEE Conf. Comput. Vis. Pattern Recog., pages 6299–6308, 2017. J. Carreira, E. Noland, A. Banki-Horvath, C. Hillier, and A. Zisserman. A short note...

    2012 arXiv

  4. [2020]

    predicted

    used as a baseline in Saxena et al. (2024). Since we are only concerned with obtaining the best possible flow model, we use a RAFT model which was trained on a large mixture of standard optical Figure 16| Examples of point trajectory radii. Circles are shown around maximum amo...

    2024

  5. [2024]

    medium” or “high

    URLhttps://github.com/hpcaitech/Open- Sora, 1(3):4. D. Zhou, W. Wang, H. Yan, W. Lv, Y. Zhu, and J. Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018, 2022. 17 Direct Motion Models for Assessing Generated Videos A. Addit...

    2018 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.