{"id":"b38c41ef-ff34-40e8-810b-918ee1a479c8","arxiv_id":"2411.18654","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"AToM uses GPT-4Vision-generated preference scores to fine-tune MotionGPT with IPO and LoRA, improving event-level alignment for integrity, temporal order, and frequency in text-to-motion generation.","lead":"This paper introduces AToM, a training pipeline that uses GPT-4Vision to score how well generated human motion clips match event-level text prompts, covering motion integrity, temporal order, and frequency, and then fine-tunes a text-to-motion model on those scores. The method shows modest metric gains over baselines and wins most head-to-head comparisons in a human study, but no code or data is released.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim relies on GPT-4V's event-level scores for integrity/temporal/frequency, but the paper never validates those scores against human ratings; the only reported 69.77% pairwise agreement is on a different task, so noisy reward labels could drive fine-tuning toward VLM heuristics.","rationale":"The paper's contribution is the RLAIF paradigm with GPT-4V reward. If the reward is unreliable, the entire mechanism collapses; the fine-tuning could just be fitting noise, and the observed improvements could stem from the LoRA tuning or the synthetic prompt distribution rather than from the reward's semantic content. The only direct evidence for reward validity is Appendix 6.4, which is disconnected from the actual scoring rubric. The reader's weakest assumption identifies the same issue, and the human study only partially mitigates it because it shares the test distribution. I do not see a reason to reject: the human study, the ablations (score filtering, LoRA, and CE-based comparison in §6.5), and the general-task improvements all suggest the reward carries some signal. The correct verdict is CONDITIONAL: the authors should be required to validate GPT-4V's per-dimension scoring against human annotations and to disclose the test-set filtering procedure. Since this is exactly the reader's condition, the verdict is unchanged.","tokens_in":15994,"tokens_out":9098,"duration_ms":81161,"concrete_test":"Manually annotate a random sample of, say, 100 prompts per sub-task from MotionPrefer (or from a fresh set generated with the same pipeline) with human raters using the exact rubrics of Table 4 (integrity: 0/5; temporal: 1-5; frequency: 1-3). Compute per-dimension exact-match accuracy and Spearman correlation between GPT-4V scores and human majority labels. If any dimension shows exact-match accuracy below ~75% (or Spearman < 0.6), the reward signal is too weak for reliable preference pairs, and the central claim that GPT-4V reward drives the improvement would be unsupported. As a secondary check, re-evaluate the fine-tuned model on an unfiltered HumanML3D test split to ensure the improvement is not limited to the filtered, synthetic-style distribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that AToM significantly improves event-level text-motion alignment presupposes that GPT-4V's reward scores (Sec 3.2, Eq 5; Table 4 rubrics) are reliable enough to create preference pairs (Algorithm 1). This is not directly supported. Appendix 6.4 reports only pairwise preference accuracy: GPT-4V agrees with human preferences on InstructMotion's pairs 69.77% of the time. That is a different task (choosing the better of two motions, presumably on general prompts) and does not validate the 1-5 (or 0/5, 1-3) event-level scoring for integrity, temporal order, and frequency. The Table 4 rubrics require counting repetitions ('three times') and judging temporal order ('then') from frames sampled every 8 frames, exactly where a VLM's visual heuristics can substitute for genuine event understanding. With scores that are only ~70% reliable on a simpler task, the constructed preference pairs will contain substantial label noise. IPO (Eq 7-8) will then amplify GPT-4V's biases rather than true event-level correctness. The human study does show AToM is preferred by humans, but it is run on the same filtered HumanML3D test set (Sec 4.1) whose filtering is not described; if that set resembles the synthetic prompt distribution used for fine-tuning, the human preference could reflect the same heuristics. Thus the central improvement claim is not yet anchored to a validated reward.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents AToM, a three-stage framework for improving event-level text-to-motion alignment. In stage one, GPT-4 constructs 5.3K synthetic prompts from HumanML3D labels, and MotionGPT generates multiple candidate motions per prompt to form the MotionPrefer dataset (Table 3). In stage two, GPT-4V scores the alignment of rendered, 8-frame-interval frame sequences against each prompt using task-specific rubrics for integrity, temporal order, and frequency (Eq. 5, Table 4). In stage three, preference pairs are constructed from score gaps (Algorithm 1) and used to fine-tune MotionGPT via IPO with LoRA (Eqs. 7–8). The authors report automated metric improvements on filtered HumanML3D test sets, human-study win rates of 70.0%–84.4% over MotionGPT (Fig. 4), and ablations on injection format, score filtering, LoRA, RL strategy, sampling interval, and dataset volume. The central claim is that GPT-4V feedback provides a scalable AI reward signal that improves event-level alignment beyond both the pretrained model and a human-feedback baseline.","tokens_in":16348,"tokens_out":4543,"duration_ms":40523,"significance":"If the central claim holds, the paper offers a practical recipe for replacing expensive human preference annotation with VLM-based reward scoring in text-to-motion alignment, and it introduces a fine-grained, multi-aspect preference dataset that could support future work. The strengths include the scale and structure of MotionPrefer, the clear ablation of motion-injection formats, the inclusion of a human evaluation, and the systematic comparison with alternative RL losses. However, the significance is currently bounded by two unverified load-bearing assumptions: that GPT-4V's event-level rubric scores are reliable enough to construct preference pairs, and that the filtered HumanML3D test set is not biased toward the synthetic prompt distribution used for fine-tuning. The paper also overstates the consistency of its quantitative gains, given that several metrics worsen or overlap within standard deviations. These issues are fixable within the manuscript's scope.","major_comments":[{"comment":"The reward signal is not validated for the exact task it is used for. GPT-4V's only reported agreement with human preference is 69.77% pairwise accuracy on InstructMotion's human-preference dataset (Appendix 6.4, Table 7), which is a binary preference task, not the 0/5, 1–5, and 1–3 event-level rubrics of Table 4 used in Eq. (5). Since Algorithm 1 and the IPO loss in Eqs. (7)–(8) rely entirely on these scores, the 69.77% figure provides no direct evidence that the constructed preference pairs are correct; if GPT-4V relies on visual heuristics when judging 8-frame-interval frame sequences, the fine-tuning could amplify those heuristics rather than improve true event-level alignment. Please validate the exact scoring rubrics against human annotations on a held-out subset of MotionPrefer, or provide an explicit noise-robustness analysis of IPO under the measured label-error rate.","section":"Sec. 3.2, Eq. (5); Appendix 6.4"},{"comment":"The filtering of the HumanML3D test set is not described. The sentence 'we filtered the HumanML3D test set, obtaining 418, 506, and 234 text-motion pairs' gives no inclusion criteria. If the filter selects prompts that match the synthetic template used to construct MotionPrefer (random HumanML3D labels joined by conjunctions), then both the automated metrics and the user study in Figure 4 are evaluated on a distribution close to the training distribution, which would inflate the reported gains and weaken the generalization claim. Please specify the filter procedure and report the overlap between the filtered test set and the synthetic prompt distribution.","section":"Sec. 4.1"},{"comment":"The claim that AToM 'consistently outperforms' baselines is too strong given the reported statistics. For integrity, FID worsens (0.400 vs. 0.340) and MM Dist and Top-1 are unchanged within standard error; MModality decreases in all three tasks (e.g., 3.495 vs. 3.846 for temporal); for frequency, Top-1/Top-2/Top-3 are essentially flat. Most reported differences are within overlapping standard deviations, including the General-task comparison against InstructMotion. Please temper the conclusion, report effect sizes or confidence intervals, and explicitly discuss the trade-offs in diversity and multimodality rather than attributing them to 'slight reductions.'","section":"Table 5; Sec. 4.2"},{"comment":"The synthetic prompt construction uses GPT-4 to join randomly selected HumanML3D labels with conjunctions, but the paper provides no evidence that this distribution resembles natural event-level prompts. Because both training and evaluation are built on this template, the approach may be fitting the template rather than improving general event-level understanding. Please include a distributional comparison between the synthetic prompts and natural event-level prompts, or a human assessment of the naturalness of the constructed prompts.","section":"Sec. 3.1, Table 3"}],"minor_comments":[{"comment":"The main text cites 'Table 12' for the temporal prompt-construction instruction, but in the supplementary material Table 12 is the frequency instruction and Table 11 is the temporal instruction; please correct the cross-reference.","section":"Sec. 3.1; Supplementary Tables 11–12"},{"comment":"Equation (6) introduces the symbol 'Dscore' but the variable defined earlier in the same section is 'Dreward'; please unify the notation.","section":"Sec. 3.2, Eq. (6)"},{"comment":"The row for 'Guo et al. [11]' lists no annotator, prompt count, or pair count, which makes the comparison incomplete; either fill in the entries or explain why the data are unavailable.","section":"Table 1"},{"comment":"The caption for Figure 5 does not identify the axes or the units, and the statement that FID has been 'negatively treated' is unexplained; please clarify the visualization.","section":"Figure 5"},{"comment":"The text says an 8-frame interval was chosen because it provides strong retrieval precision, but Figure 6 shows lower FID at interval 4; please justify the choice or acknowledge the trade-off.","section":"Sec. 4.3, Figure 6"},{"comment":"The term 'event-level' is used repeatedly but is never formally defined; a brief definition in Section 1 would improve precision and help readers connect the three sub-tasks to the concept.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely problem and the dataset/ablations are useful, but the two load-bearing gaps—reward validation and test-set filtering—prevent a reliable assessment of the central claim. Both are fixable with additional experiments and analysis, so I recommend major revision rather than rejection. The scope is appropriate for a vision conference, and the presentation is mostly clear."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things: the paper introduces a genuinely new reward source for text-to-motion alignment, and the central claim is not yet anchored to a validated reward. The authors use GPT-4V to score generated motions against event-level prompts on three dimensions—integrity, temporal order, frequency—build a large preference dataset (5.3K prompts, 80K pairs), and fine-tune MotionGPT with LoRA and IPO. That combination is new relative to prior human-feedback (InstructMotion) or contrastive-encoder (MoDiPO, Mao et al.) approaches. The human study is the strongest evidence: win rates of 70–84% over the pretrained model across tasks, which is credible support that the fine-tuning improves perceived alignment. The ablation on motion injection forms is also well done.\n\nNow the soft spots. The reward model is the load-bearing component, and the paper never validates GPT-4V's event-level scores against human judgments on those specific scoring rubrics. The only reported agreement, 69.77% on InstructMotion's pairwise human-preference data, is a different task—choosing the better of two motions, not rating integrity/temporal/frequency. Given the rubrics require counting repetitions and judging order from 8-frame-interval renderings, the constructed preference pairs likely contain substantial label noise. IPO will amplify whatever biases GPT-4V has, and the human study's use of the same filtered test set (whose filtering is undescribed) could reflect the same heuristics. The quantitative results are also mixed: integrity FID worsens (0.400 vs 0.340), MModality drops across all tasks, and most metric gains are within overlapping standard deviations. Calling these gains \"significant\" is an overstatement. The paper also tunes several hyperparameters (β, score threshold, frame interval, epochs) on what appear to be the same test metrics, and no code or data is released.\n\nStill, these are concerns about proof strength, not about coherence. The pipeline is sensible, the dataset is a real contribution, and the idea that a general VLM can provide fine-grained alignment rewards for motion is worth testing and refining. The authors are making a reasonable empirical bet, and the human preference results suggest something is working.\n\nThis paper is for researchers working on text-to-motion alignment, preference-based fine-tuning, and RLAIF for generative models. It deserves a serious referee: the novelty is real and the topic matters. The review should ask for validation of the reward on the actual scoring tasks, disclosure of the test-set filtering, and release of code/data. If the authors can address reward reliability, acceptance is defensible; as written, it is conditional.\n\nRecommendation: send to peer review with a request for major revision. The flaws are fixable and the core idea is worth engaging.","headline":"Genuinely new reward source for text-to-motion event-level alignment, but the reward model is not validated on the actual scoring tasks and the quantitative gains are modest; still deserves a serious referee.","tokens_in":16888,"tokens_out":1520,"would_cite":true,"duration_ms":15079,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AToM fine-tunes a text-to-motion model on GPT-4V-scored preference data to improve event-level alignment across integrity, temporal order, and frequency.","keywords":["text-to-motion generation","event-level alignment","GPT-4V reward","preference optimization","AI feedback","MotionPrefer","motion-text alignment","LoRA fine-tuning"],"falsifier":"Take a held-out set of event-level prompts with ground-truth annotations for which motions appear, in what order, and how many times; generate motions; then compare AToM against the base model on the subset where GPT-4V's scores disagree with human labels. If AToM's advantage disappears or reverses there, the GPT-4V reward is the weak link. An even sharper test is to retrain the same pipeline with GPT-4V scores replaced by random or inverted scores: if the reported metric gains persist, they are not caused by the reward's semantic content.","tokens_in":15796,"feed_emoji":"🏃","tokens_out":7358,"duration_ms":64963,"temperature":0.7,"pith_summary":"This paper proposes that a vision-language model's scores of rendered motions can act as a training signal for text-to-motion generation, without motion-capture annotation or human preference labeling. It builds MotionPrefer, a dataset of 5,276 event-level prompts and 80K motion preference pairs, where each motion is graded by GPT-4V on whether it contains all described actions, puts them in the right order, and repeats them the right number of times. Fine-tuning MotionGPT on this data with LoRA and an IPO preference loss is reported to improve event-level alignment across all three dimensions, beating both the base model and the human-feedback baseline InstructMotion on most metrics. A human study backs the result, with AToM preferred in 70 to 84 percent of comparisons depending on the task. If the finding holds, scalable AI feedback can replace costly human labels for fine-grained motion-text alignment.","feed_headline":"GPT-4V reward teaches text-to-motion models event-level correctness","feed_subtitle":"Fine-tuning MotionGPT on 80K GPT-4V-scored pairs boosts integrity, temporal order, and frequency alignment.","key_machinery":"The load-bearing object is MotionPrefer, a preference dataset built from event-level prompts and GPT-4V scores, together with the reward paradigm that produces those scores. Central to the paradigm is the motion injection format: generated 3D motion is rendered and sampled as a sequence of frames spaced eight apart, which GPT-4V sees alongside the prompt and a task-specific scoring rubric. The fine-tuning stage converts these scores into preference pairs by grouping motions by prompt, ranking them by score, and keeping only pairs whose score gap exceeds a threshold. The model is then updated with the IPO loss, a preference-optimization objective that trains the policy directly on the ranked pairs, combined with LoRA low-rank adaptation for parameter-efficient tuning.","core_discovery":"The paper's central claim is that event-level alignment failures, such as missing motion events, wrong temporal order, and wrong repetition counts, can be corrected by reward fine-tuning on GPT-4V judgments. The authors generate motions from constructed prompts, render them to video, sample frames every eight frames, and ask GPT-4V to score each motion-prompt pair under task-specific rubrics: integrity on a 0/5 scale, temporal order on a 1 to 5 scale, and frequency on a 1 to 3 scale. These scores form the MotionPrefer dataset, from which preference pairs are selected only when score differences exceed a threshold. Fine-tuning MotionGPT with LoRA and the IPO loss on these pairs is shown to lower text-motion distance, raise retrieval precision, and improve FID relative to MotionGPT and InstructMotion, with human win rates of 84.4 percent for integrity, 74.4 percent for temporal order, and 70.0 percent for frequency.","pith_inferences":["If GPT-4V scoring is the bottleneck, as the paper's own 69.77 percent agreement with human labels suggests, then augmenting or replacing it with a small human-labeled reward model could push the same pipeline further.","The render-and-score preference loop could generalize to other generative domains whose outputs can be visualized, such as text-to-video, audio spectrograms, or 3D scene generation, whenever event-level correctness matters.","The visible effects of score filtering and preference dataset volume imply that data curation quality may matter more than the choice of RL algorithm, so a cleaner larger dataset could outperform a more sophisticated loss.","The diversity and multimodality trade-off visible in the metrics points to a potential failure mode: over-optimizing the reward could collapse the motion distribution, making a diversity-regularized variant a natural next experiment."],"forward_implications":["Event-level alignment can be improved without new motion-capture data: the reward comes from rendered images and a frozen vision-language model.","The same reward paradigm can be reused with other vision-language models as they improve, since only the scoring instructions would need to change.","Fine-grained preference signals covering integrity, order, and frequency isolate which failure mode a motion model still has, making the dataset a diagnostic tool as well as a training set.","The approach offers a scalable alternative to human RLHF data collection, reducing the cost of aligning text-to-motion models to detailed event descriptions.","The reported reductions in diversity and multimodality indicate that alignment gains come with a coverage trade-off that downstream motion applications will need to manage."],"supporting_citations":[{"why":"Provides MotionGPT, the off-the-shelf text-to-motion model that AToM fine-tunes and compares against as the base generator.","marker":"[15]"},{"why":"Supplies InstructMotion, the human-preference RLHF baseline that AToM must beat, and the human preference dataset used in the agreement check.","marker":"[29]"},{"why":"Identifies GPT-4V as the vision-language model that produces the reward scores for MotionPrefer.","marker":"[21]"},{"why":"Defines the IPO preference-optimization loss used to fine-tune the motion generator on scored preference pairs.","marker":"[6]"},{"why":"Provides LoRA, the low-rank adaptation method used for parameter-efficient fine-tuning in AToM.","marker":"[14]"},{"why":"Supplies the HumanML3D motion-event labels used to construct prompts and the filtered test sets for evaluation.","marker":"[11]"},{"why":"Represents the contrastive text-motion encoder reward baseline that AToM compares against in the supplementary experiments.","marker":"[20]"},{"why":"Provides a prior AI-feedback preference dataset for text-to-motion alignment, used as context for the MotionPrefer comparison.","marker":"[23]"}],"fun_headline_variants":["GPT-4V reward aligns text-to-motion at event level","AToM: event-level motion alignment via GPT-4V reward","Fine-tune motion with GPT-4V scores for event correctness","Reward from GPT-4V fixes integrity, order, and frequency","GPT-4V-scored pairs teach motion models event-level alignment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole gain rests on GPT-4V being a trustworthy judge of event-level alignment from a handful of still frames, even though the paper's own check puts its agreement with human labels at only 69.77 percent.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4V reward aligns text-to-motion at event level","AToM: event-level motion alignment via GPT-4V reward","Fine-tune motion with GPT-4V scores for event correctness","Reward from GPT-4V fixes integrity, order, and frequency","GPT-4V-scored pairs teach motion models event-level alignment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000206,"raw_usage":{"total_tokens":1384,"prompt_tokens":922,"completion_tokens":462,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":369}},"tokens_in":538,"tokens_out":462,"duration_ms":4475,"temperature":1.0,"reasoning_tokens":369,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:32:14.659423+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of event-level prompts with ground-truth annotations for which motions appear, in what order, and how many times; generate motions; then compare AToM against the base model on the subset where GPT-4V's scores disagree with human labels. If AToM's advantage disappears or reverses there, the GPT-4V reward is the weak link. An even sharper test is to retrain the same pipeline with GPT-4V scores replaced by random or inverted scores: if the reported metric gains persist, they are not caused by the reward's semantic content.","supporting_citations":[{"cited_title":"Motiongpt: Human motion as a foreign lan- guage","cited_arxiv_id":null,"evidence_quote":"Provides MotionGPT, the off-the-shelf text-to-motion model that AToM fine-tunes and compares against as the base generator."},{"cited_title":"Exploring text-to-motion generation with human preference","cited_arxiv_id":null,"evidence_quote":"Supplies InstructMotion, the human-preference RLHF baseline that AToM must beat, and the human preference dataset used in the agreement check."},{"cited_title":"Gpt-4 technical report, 2024","cited_arxiv_id":null,"evidence_quote":"Identifies GPT-4V as the vision-language model that produces the reward scores for MotionPrefer."},{"cited_title":"A general theoretical paradigm to un- derstand learning from human preferences","cited_arxiv_id":null,"evidence_quote":"Defines the IPO preference-optimization loss used to fine-tune the motion generator on scored preference pairs."},{"cited_title":"Generating diverse and natural 3d human motions from text","cited_arxiv_id":null,"evidence_quote":"Supplies the HumanML3D motion-event labels used to construct prompts and the filtered test sets for evaluation."}],"review_version":1}