{"id":"b5a2e131-b0a4-46e4-b416-4c2601d71fc8","arxiv_id":"2505.00209","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"TRAJAN, an autoencoder trained on point tracks, provides a video-quality metric that is more sensitive to motion distortions and better aligned with human ratings than existing video metrics.","lead":"This paper introduces TRAJAN, a method that converts point trajectories in videos into a compact learned representation and scores motion quality in AI-generated videos by how well those trajectories can be reconstructed. It reports that TRAJAN detects temporal distortions and matches human judgments of realism better than existing metrics, while also scoring single videos without any reference.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The single-video metric rests on an untested premise that unrealistic motions are harder to reconstruct, and Figure 6 is a paper-internal counterexample (a glass collapsing smoothly scores positively); the abstract's claim that TRAJAN evaluates single-video motion is therefore not yet established.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing premise: that reconstruction error on real-world-trained autoencoders orders generated videos by perceptual motion quality. I agree with that identification, and the paper's own Figure 6 strengthens it from a hypothetical to an observed counterexample: a physically implausible but smooth and trackable motion receives a positive TRAJAN score. This matters because the single-video mode is a headline contribution, explicitly contrasted with FVD-style distributional metrics. The paper's evidence for the premise is indirect: the UCF elastic-transformation experiment shows sensitivity to synthetic temporal distortions that degrade trackability by design, and the human correlations (Tables 2, 6) are modest, unreported with confidence intervals, and obtained with a loss weight tuned on such correlations. None of this independently establishes that AJ measures motion realism across the range of generated-video failures. I do not see this as a fatal flaw: the comparative claims (better than alternatives at predicting human ratings, high sensitivity to temporal distortions) remain supported by the reported numbers, and the authors openly acknowledge the glass-collapse failure. The right outcome is unchanged from the reader's CONDITIONAL: the paper's central comparative claims are plausible and empirically grounded, but the single-video interpretation requires a direct test of the reconstruction-error premise, plus a release of confidence intervals or a pre-registered holdout evaluation. My concrete test would settle whether the premise holds or needs to be narrowed to 'hard-to-track motion' rather than 'unrealistic motion'.","tokens_in":25370,"tokens_out":5214,"duration_ms":56473,"concrete_test":"Use a synthetic video generator with known ground-truth motion (e.g., Kubric) to create matched pairs of videos that are visually similar and equally trackable but differ in physical plausibility: for example, an object moving with constant velocity versus the same object undergoing sudden nonphysical acceleration or an instantaneous, smooth-but-impossible morph. Run TRAJAN's single-video mode (Average Jaccard) on each pair, using both BootsTAPIR-estimated tracks and the simulator's ground-truth tracks. If AJ does not systematically rank the physically implausible videos as worse (e.g., a paired Wilcoxon test fails to reject at a pre-specified effect size), then the core premise of Section 3.1 is falsified for smooth unrealistic motions and the single-video claim in the abstract must be qualified accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract claims TRAJAN can evaluate the motion of single videos, and Section 3.1 justifies the per-video Average Jaccard score by stating: \"the model was trained on real motions, so we expect unrealistic motions to be more difficult to reconstruct.\" This premise is load-bearing for the single-video mode (Section 5.3) but is never tested independently of the human-rating correlations or the UCF elastic-transformation experiment. The paper itself provides a counterexample: in Figure 6, a beer glass collapses into a table — physically implausible, but the motion is smooth and trackable — and TRAJAN, like all metrics, scores it positively. The conclusion explicitly concedes that no automated metric captures such cases. Additionally, AJ is computed on BootsTAPIR tracks; on generated videos with morphing or disappearing objects, tracker failures can inflate reconstruction error regardless of whether the underlying motion is unrealistic, confounding the metric's interpretation. Because the single-video contribution is a central claimed benefit over FVD-style methods, the unsupported premise means this mode is not yet validated as a general measure of motion realism. The distribution-level sensitivity results and the human correlations are real evidence, but they do not establish the mechanism claimed for per-video scoring; the correlations are modest and the synthetic distortions are, by construction, cases where tracking is degraded.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TRAJAN, a trajectory autoencoder trained on point tracks extracted by BootsTAPIR, and uses its latent space or reconstruction error (Average Jaccard) as a video quality metric. The authors claim three modes of use: distribution-level comparison via Fréchet distance, paired-video comparison via latent L2 distance, and single-video evaluation via reconstruction error. They report that TRAJAN is markedly more sensitive to synthetic temporal distortions on UCF-101 than appearance-based and histogram-based alternatives, and that its per-video scores correlate with human ratings of consistency, realism, interaction quality, and speed on EvalCrafter and VideoPhy, outperforming a range of baselines. They also introduce a new human-study protocol and a spatiotemporal error localization tool.","tokens_in":25591,"tokens_out":4430,"duration_ms":48183,"significance":"If the claims hold, TRAJAN would be a useful contribution to video generation evaluation, particularly for distribution-level sensitivity to temporal distortions and for the interpretability of localized motion inconsistencies. The synthetic UCF-101 experiment is a clean and convincing demonstration of motion sensitivity, and the human study is a substantial empirical contribution covering two public datasets and multiple rating dimensions. The promise of code release and the detailed experimental appendix are strengths. However, the central per-video claim rests on an untested mechanism, the paired-video mode lacks quantitative validation, and the human-correlation evidence is modest and possibly affected by model selection on human labels. These issues currently prevent accepting the paper's strongest claims at face value.","major_comments":[{"comment":"The per-video metric rests on the premise stated in Section 3.1 that, because TRAJAN was trained on real motions, unrealistic motions will be more difficult to reconstruct. This premise is never tested independently of the human-rating correlations or the synthetic UCF-101 experiment. The paper itself provides a counterexample in Figure 6, where a physically implausible glass collapse is scored positively by all metrics. Moreover, because Average Jaccard is computed against BootsTAPIR tracks, failures of the tracker on generated videos with morphing or disappearing content will inflate reconstruction error regardless of whether the underlying motion is unrealistic, confounding the interpretation of the metric. I request either a direct test of the mechanism (e.g., perturbing motion while controlling trackability) or a careful narrowing of the claim to what the evidence supports, namely that AJ captures smoothness or trackability rather than physical realism.","section":"Section 3.1 and Section 5.3"},{"comment":"The description of TRAJAN training discloses that the Huber and occlusion loss weights were selected in initial experiments by their correlation with human judgements of realism in generated videos. Since the headline result in Section 5.3 is that TRAJAN predicts human ratings better than alternatives, this creates a fairness problem: TRAJAN is tuned with respect to the evaluation target, while the comparison methods (VideoMAE, I3D, RAFT, MooG) are used as off-the-shelf. The authors should either report TRAJAN with default or equally weighted losses, or quantify how much of the advantage in Table 2 depends on this human-label-based selection.","section":"Appendix B.2.1"},{"comment":"The paired-video mode is validated only by showing low correlation with PSNR and SSIM and by qualitative examples. Low correlation with pixel-based metrics establishes that TRAJAN measures something different from pixel error, but it does not establish that the latent distance correctly captures similarity in motion. Without a controlled experiment or human judgments for paired comparisons, the abstract's claim that TRAJAN can compare one generated video to one ground truth video is not supported. Please add a quantitative validation of the paired-video mode, for example by using synthetic motion-preserving or motion-degrading perturbations and measuring whether the ranking of pairs matches ground truth.","section":"Section 5.2 and Table 1"},{"comment":"The human-rating correlations are modest (Spearman coefficients of approximately 0.09 to 0.32 in Table 2, and 0.24 to 0.33 in the motion-controlled analysis of Table 4), yet the conclusion states that reconstruction error 'correlates remarkably well' with human judgments. No confidence intervals or significance tests are reported, and the inter-rater standard deviations are high (around 0.5). For several cells the advantage over baselines is small or tied (e.g., VideoPhy Interacts: TRAJAN 0.09 versus I3D 0.09). The authors should report confidence intervals, compare effect sizes with rater noise, and soften the conclusion accordingly.","section":"Tables 2 and 4"}],"minor_comments":[{"comment":"There is a typo in the first bullet list: 'synthethic' should be 'synthetic'.","section":"Section 1"},{"comment":"The phrase 'as few as one generated and one ground truth' in the distribution-level comparison setting is confusing, since a Fréchet distance between two one-sample distributions is just the L2 distance between two embeddings. Please clarify what is meant.","section":"Abstract"},{"comment":"The sentence 'with average points within threshold of 85.3' should presumably read '85.3%' to indicate a percentage of points within the threshold.","section":"Appendix B.2.1"},{"comment":"The left panel of Figure 4 would benefit from error bars or a description of how the 'average temporal sensitivity' is computed across corruption levels, since the main distribution-level claim depends on this quantity.","section":"Figure 4"},{"comment":"In Table 1, the interpretation that 'distances in latent space are not well correlated with PSNR or SSIM' would be clearer if the authors stated explicitly that this is evidence of orthogonality rather than of correctness.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically strong in its distribution-level experiments and in the breadth of the human study, but the per-video and paired-video claims are currently overextended relative to the evidence. The disclosure in Appendix B.2.1 that TRAJAN's losses were tuned on human realism judgments is a fairness concern for the headline comparison. A revision that adds a direct test of the reconstruction premise, a fairer comparison protocol, and confidence intervals for the human correlations would address the main risks. I think major_revision is appropriate rather than rejection, because the issues are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a real advance. The idea of auto-encoding point tracks into a fixed-size latent and using reconstruction error as a per-video motion score is new relative to the cited histogram and appearance-based alternatives, and the paper backs it with a sensible set of experiments. The synthetic UCF-101 elastic-distortion test cleanly separates TRAJAN from FVD-style baselines, and the human studies on EvalCrafter and VideoPhy show TRAJAN correlates with ratings at least as well as RAFT warp error and better than VideoMAE/I3D/MooG across most dimensions. The spatiotemporal error localization is a nice bonus that no other metric in this comparison offers.\n\nThe soft spots are real but not fatal. The human correlations are modest (Spearman roughly 0.1–0.3) and reported without confidence intervals, which matters when the differences between TRAJAN and RAFT are not huge. The pairwise video-video mode is validated only indirectly: the paper shows TRAJAN distances are uncorrelated with PSNR/SSIM, which establishes it is not just pixel error, but there is no human-judgment test for that mode. The per-video premise that unrealistic motions are harder to reconstruct is plausible and is supported indirectly by the human correlations, but the paper does not isolate this mechanism from a simpler explanation: tracker failures on generated videos might inflate reconstruction error regardless of motion plausibility. Figure 6, where a smoothly collapsing glass is scored positively, is a concrete case where the premise fails, and the paper honestly concedes that no automated metric handles such cases. The loss-weight tuning on human-rating correlation is a minor circularity, not a fatal one, because the main representations are learned on unlabeled real videos.\n\nOverall, this is a solid, useful paper for anyone working on video generation evaluation. It deserves a serious referee: the experiments are reproducible in spirit, the code is linked, and the limitations are stated rather than hidden. The referee should ask for confidence intervals, a sensitivity analysis around tracker failures, and a clearer validation of the pairwise mode. I would cite this and would bring it to reading group.","headline":"TRAJAN is a genuinely useful new motion-quality metric for generated videos; the distribution-level and human-correlation evidence holds up, but the single-video mode rests on a premise the paper only partially validates.","tokens_in":26248,"tokens_out":1495,"would_cite":true,"duration_ms":17057,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that evaluating generated videos by auto-encoding point trajectories — rather than comparing pixels or action-recognition features — yields a motion metric that is markedly more sensitive to temporal distortions and…","keywords":["point tracks","video generation evaluation","trajectory autoencoder","temporal consistency","Fréchet video distance","Average Jaccard","human evaluation of video quality","generative video models"],"falsifier":"Compare TRAJAN's Average Jaccard on a set of real videos containing unusual but physically realistic motion (e.g., cloth flapping, water splashes, or slow-motion impacts) against ordinary action videos: if the unusual-but-real videos systematically receive low reconstruction scores while humans judge them realistic, the per-video ordering claim collapses. Alternatively, take a single real video and re-render it with the same motion but different textures; if Average Jaccard changes substantially, the metric is not purely motion-based.","tokens_in":25121,"feed_emoji":"🎬","tokens_out":7132,"duration_ms":63935,"temperature":0.7,"pith_summary":"The paper sets out to fix a blind spot in how generated videos are judged: models can produce frames that look fine but move wrongly, and standard metrics such as FVD miss this because they lean on appearance and action-recognition features. The authors propose TRAJAN, a trajectory autoencoder that compresses a video's point tracks into a fixed latent vector and reconstructs held-out tracks from that vector. Because the model is trained only on real-world motion, its reconstruction error—measured by Average Jaccard—acts as a per-video score for motion plausibility, while Fréchet distance over the latents compares whole datasets and L2 distance compares video pairs. On synthetic temporal-distortion tests and on human ratings for videos from open-source generators, TRAJAN outperforms I3D, VideoMAE, motion histograms, optical-flow warping, and next-frame prediction baselines. If the paper's central claim holds, video quality evaluation need not depend on a reference dataset or on expensive human studies.","feed_headline":"Point tracks outrank FVD for judging video motion","feed_subtitle":"A trajectory autoencoder predicts human ratings of motion quality without needing a reference dataset.","key_machinery":"The central object is the TRAJAN trajectory autoencoder operating on point tracks. Point tracks are temporally extended (x, y, occlusion) trajectories of arbitrary points, extracted by BootsTAPIR; they separate motion from appearance because they never reconstruct pixels. TRAJAN embeds each track with sinusoidal position encodings, summarizes it with self-attention using occlusion flags as masks, and compresses the whole set with a Perceiver-style cross-attention into 128×64 latent tokens. A decoder consumes a query point (xq, yq, tq) and predicts the full track through that point; training on held-out queries forces the latent to capture a dense motion field. Average Jaccard, borrowed from TAP-Vid, measures reconstruction accuracy across pixel thresholds and serves as the per-video quality score.","core_discovery":"The central claim is that point tracks are a better substrate than pixels or action-recognition features for measuring motion quality in generated video, and that an autoencoder trained on those tracks can serve as a single unified metric across all three evaluation settings. TRAJAN encodes a variable-sized, orderless set of point trajectories—extracted by BootsTAPIR—into a fixed 128×64 latent using a Perceiver-style transformer, and is trained to reconstruct query tracks that were held out from the input, making the representation invariant to which points were sampled and to occlusion. The same model then yields a distribution metric (Fréchet distance between latent Gaussian fits), a pairwise metric (L2 distance between latents), and an individual-video metric (Average Jaccard reconstruction error). The paper reports that the Fréchet variant is more than three times as sensitive to synthetic temporal distortions as prior backbones, that the latent distance ranks WALT-generated videos near their real counterparts when motion is correct even if pixels diverge, and that the reconstruction score correlates with human ratings of consistency, realism, and interactions better than all tested alternatives on EvalCrafter and VideoPhy data, while remaining competitive on optical-flow warping.","pith_inferences":["If realistic motion is indeed easier to reconstruct than unrealistic motion, the same autoencoder could be repurposed as a differentiable reward or loss for training or fine-tuning generative video models, not just an evaluation metric.","The finding that human raters disagree substantially with one another suggests per-video quality is partly subjective; a practical direction is to build a metric that predicts a distribution of human ratings rather than a single scalar, or to calibrate any metric against rater agreement.","The paper's motion-only distance could be combined with prompt-semantic metrics such as CLIPScore to yield a two-axis evaluation: adherence to the text and physical plausibility of the motion.","One testable extension is to apply TRAJAN to unusual but physically realistic motion—fluids, cloth, or slow motion—where reconstruction difficulty may not track perceived realism; if it fails there, the ordering claim is narrower than human perception."],"forward_implications":["On the UCF-101 elastic-corruption test, the TRAJAN Fréchet distance is markedly more sensitive to temporal distortions than FVD backbones, and the per-video Average Jaccard nearly matches the distribution-level sensitivity, suggesting per-video scores can substitute for reference-based comparisons when no reference is available.","A reference-free per-video motion score means one can rank individual generated videos, monitor training checkpoints, and filter outputs without needing the generator's training distribution, which FVD requires.","TRAJAN's latent distance ranks generated WALT videos that share motion with the real continuation as closer than appearance-based latent distances do, even when pixel error is large, indicating a motion-only similarity signal.","The reconstruction error correlates with human ratings across multiple dimensions besides motion—appearance consistency, realism, and object interactions—and tracks which generator humans prefer overall, suggesting motion quality carries much of the perceived-quality signal.","Because Average Jaccard can be computed per point and per frame, TRAJAN localizes where and when generative inconsistencies occur, as shown for morphing hands and gloves."],"supporting_citations":[{"why":"BootsTAPIR: supplies the point tracks that TRAJAN encodes and reconstructs.","marker":"(Doersch et al., 2024)"},{"why":"TAP-Vid: defines Average Jaccard and the pixel thresholds used for reconstruction scoring.","marker":"(Doersch et al., 2022)"},{"why":"Introduces FVD and the Fréchet-distance protocol for comparing video distributions, the paradigm TRAJAN extends.","marker":"(Unterthiner et al., 2018)"},{"why":"Shows FVD's content bias and provides the elastic-corruption protocol and temporal-sensitivity ratio used to test motion sensitivity.","marker":"(Ge et al., 2024)"},{"why":"I3D: the action-recognition backbone behind FVD, used here as an appearance-based baseline.","marker":"(Carreira and Zisserman, 2017)"},{"why":"VideoMAE v2: the masked-autoencoder alternative used as an appearance-based feature and reconstruction baseline.","marker":"(Wang et al., 2023b)"},{"why":"Motion histograms: the prior track-based metric that TRAJAN is compared against and markedly outperforms.","marker":"(Liu et al., 2024a)"},{"why":"EvalCrafter: source of generated videos and human ratings used to validate per-video and per-model predictions.","marker":"(Liu et al., 2024c)"},{"why":"VideoPhy: benchmark with human physical-consistency labels used for ROC-AUC validation and per-video correlations.","marker":"(Bansal et al., 2024b)"},{"why":"Perceiver: the cross-attention architecture TRAJAN uses to compress a variable set of tracks into fixed latents.","marker":"(Jaegle et al., 2021)"}],"fun_headline_variants":["Point tracks beat pixels for video motion metrics","TRAJAN: track autoencoder outperforms FVD on motion","Track-based metric triple-senses temporal warping","One autoencoder, three motion checks: tracks win","Autoencoded point tracks predict human video quality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that an autoencoder trained on ordinary real-world point tracks will find realistic motion easier to reconstruct than unrealistic motion, so its reconstruction error ranks videos by perceptual quality — and the paper validates this only indirectly through correlations with human ratings, not by independently confirming that unusual-but-real motions are reconstructed well.","fun_headline_variants_meta":{"raw":{"variants":["Point tracks beat pixels for video motion metrics","TRAJAN: track autoencoder outperforms FVD on motion","Track-based metric triple-senses temporal warping","One autoencoder, three motion checks: tracks win","Autoencoded point tracks predict human video quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000189,"raw_usage":{"total_tokens":1351,"prompt_tokens":975,"completion_tokens":376,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":301}},"tokens_in":591,"tokens_out":376,"duration_ms":4626,"temperature":1.0,"reasoning_tokens":301,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:48:45.388442+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare TRAJAN's Average Jaccard on a set of real videos containing unusual but physically realistic motion (e.g., cloth flapping, water splashes, or slow-motion impacts) against ordinary action videos: if the unusual-but-real videos systematically receive low reconstruction scores while humans judge them realistic, the per-video ordering claim collapses. Alternatively, take a single real video and re-render it with the same motion but different textures; if Average Jaccard changes substantially, the metric is not purely motion-based.","supporting_citations":[],"review_version":1}