{"id":"0a0f212e-0877-49c2-bcdf-418e99d477f2","arxiv_id":"2509.01471","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"This paper introduces a hierarchical motion captioning system that generates low-level descriptions with an LLM and retrieves high-level captions from a database, reporting large gains over prior methods on three datasets.","lead":"This paper proposes a two-step motion captioning method that first uses a large language model to write detailed low-level descriptions of movements, then retrieves matching high-level captions from a database to generate the final caption. The authors report gains of 6% to 50% over prior methods on three motion-text datasets, though the claimed use of external text data is not actually tested.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's headline claim about external text sources is not tested: the only database-enrichment experiment adds validation captions from the same datasets, not standalone external text, and no rare/unseen-motion evaluation is reported.","rationale":"The reader's weakest_assumption focuses on unverified Falcon-40B labels and cosine-similarity reliability. Both are legitimate concerns, but I see a more decisive gap: the paper's advertised contribution—using external text sources—is never operationalized in the experiments. The only enrichment variant (HiCAM2T+) adds validation-set captions from the same benchmarks; those captions are motion-paired, in-distribution, and not 'external' in the sense of standalone text. Moreover, the paper reports no evaluation on categories absent from the training database, so the 'especially for movements not covered' part of the claim is unsupported. This is not an internal inconsistency; the architecture could plausibly work if given external entries. But the absence of any such test means the central claim cannot be accepted as stated. The ablations (Table 6) and benchmark tables do support the hierarchical retrieval framework within the tested setup, so the verdict should remain conditional: require an external-text experiment before accepting the headline generalization. Agreement with the reader is partial: the reader lists the external-source assumption as secondary; I make it primary.","tokens_in":13541,"tokens_out":6378,"duration_ms":77509,"concrete_test":"Construct an external-text condition for HumanML3D: split actions so a held-out set of test categories (e.g., all motions whose captions contain 'soccer', 'dance', 'throw') are absent from the training set and the training-only database. Add only text-only captions for those categories (e.g., high-level captions from an external corpus or LLM-generated descriptions not paired with motion) to the database. Compare BLEU-4 and CIDEr on the held-out set with and without the external entries, and also against the current HiCAM2T+ validation-enrichment condition. If external entries do not improve performance (or do not beat validation-set enrichment), the central external-text claim fails. A secondary check: report retrieval precision@k on a labeled subset of low-level descriptions to verify the cosine-similarity retrieval assumption.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim—'harness knowledge from external text sources to greatly increase motion captioning accuracy, especially for movements not covered in existing motion-text datasets'—requires that adding text-only entries to the retrieval database (Sec. 3.4) improves captioning for motions or action categories absent from the training data. No experiment tests this. The only database-enrichment comparison (Sec. 4.2) is HiCAM2T vs. HiCAM2T+, where 'the latter uses an enriched database that encompasses motion captions from both the training set and the validation set.' The validation captions are paired with motions from the same datasets, not external/standalone text, and they are not a proxy for 'movements not covered in existing motion-text datasets.' The paper never adds captions from outside the dataset, never evaluates a held-out category absent from the training database, and reports no retrieval-quality or per-category results. Consequently, the distinguishing novelty—external text—is unverified; the reported gains could come entirely from within-dataset retrieval augmentation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HiCAM2T, a hierarchical motion-captioning model. A ViT motion encoder and a GPT-2-family text decoder are first trained to generate low-level descriptions of a motion; these descriptions are encoded with Sentence-BERT and used to retrieve the top-k high-level captions from a database of (low-level, high-level) caption pairs. The retrieved captions and motion features are then concatenated and passed through a text decoder to produce the final high-level caption. The method is evaluated on KIT, HumanML3D, and BOTH57M, reporting large gains over T2MT, M2T-INT, MotionGPT, LaMP, and MotionLLM on BLEU/ROUGE/CIDEr, and ablations support the individual components. The abstract and introduction further claim that the method harnesses external text sources and is especially beneficial for movements not covered in existing motion-text datasets.","tokens_in":13842,"tokens_out":8178,"duration_ms":91778,"significance":"If the external-text claim were experimentally established, the contribution would be practically important: it would decouple motion captioning from expensive motion-capture data collection and allow text-only expansion of captioning coverage. The paper's strengths are its clear algorithmic presentation (Algorithm 1), the multi-dataset evaluation, the consideration of both lemmatized and non-lemmatized settings, and the candid limitation discussion. However, the headline innovation—external text sources and generalization to unseen movements—is not tested: the only enrichment experiment adds validation captions from the same datasets, and the low-level captions that drive both training and retrieval are largely unverified. The current evidence supports a retrieval-augmented captioning system with strong in-distribution performance, but not the 'external knowledge' claim as stated.","major_comments":[{"comment":"The abstract and introduction claim that the method 'harness[es] knowledge from external text sources' and is especially beneficial for 'movements not covered in existing motion-text datasets.' The only database-enrichment experiment compares HiCAM2T with HiCAM2T+, where the latter uses 'motion captions from both the training set and the validation set.' These are paired captions from the same datasets, not standalone external text, and no held-out action category or rare-motion evaluation is reported. The distinguishing novelty is therefore unsupported. Please add experiments with genuinely external text-only entries (e.g., action descriptions without paired motion data) and report performance on rare/unseen categories, or revise the abstract and introduction to describe the contribution as within-dataset retrieval augmentation.","section":"Sec. 3.4 and Sec. 4.2"},{"comment":"The paper states that fewer than 50 of tens of thousands of Falcon-generated low-level captions were verified. These captions are used as the ground-truth target for the low-level generation loss (Eq. 4) and as the retrieval database entries (Sec. 3.4). If these synthetic labels are systematically inaccurate, both the motion encoder and the retrieval encoder are trained on incorrect supervision. The analogy to noisy image-text pairs does not address the fact that here the synthetic text is the sole supervisory signal for a novel intermediate representation that the whole method depends on. Please provide a human-verified sample evaluation, an automated noise analysis, or an explicit robustness experiment to noisy low-level labels.","section":"Sec. 3.3 and Eq. (4)"},{"comment":"The retrieval mechanism itself is never directly evaluated. The method assumes that cosine similarity between Sentence-BERT embeddings of low-level descriptions is a reliable proxy for action identity (Eqs. 5-6), but no retrieval-quality metrics (e.g., recall@k against ground-truth caption identity), qualitative retrieval examples, or per-category results are reported. A reader cannot tell whether the improvements come from retrieving semantically relevant captions or from the decoder mostly ignoring the retrieval. Please report retrieval accuracy/recall@k and, ideally, error analysis for rare or ambiguous motions.","section":"Sec. 3.1, Eq. (5)-(6), Sec. 4.2"},{"comment":"The enrichment claim is not consistently supported. On KIT (Table 3), HiCAM2T+ has lower ROUGE-L (60.4 vs. 63.3) and lower CIDEr (169.8 vs. 203.0) than HiCAM2T, and the average change is negative. The introduction's 'up to a 13% increase in average performance' from enrichment appears to come from a subset of the settings. Please report the exact averaging procedure over metrics and include per-dataset summaries or confidence intervals so that the strength of the enrichment claim is transparent.","section":"Sec. 4.2 and Sec. 4.3"}],"minor_comments":[{"comment":"The sentence 'The detailed explanation generated by the text encoder is processed by a text decoder' appears to conflate encoder and decoder; the subsequent equations use a text encoder (TE) to produce embeddings. Please clarify.","section":"Sec. 3.1"},{"comment":"The notation is inconsistent: Equation (6) uses \\hat{u} while Algorithm 1 line 15 uses \\hat{u}u. Use the same symbol, e.g., \\hat{u}, throughout.","section":"Eq. (6) and Algorithm 1"},{"comment":"The text says the optimal k is 2 for KIT and BOTH57M, but Table 5 shows k=1 is best for BOTH57M on all four metrics, and the hyper-parameters in Sec. 4.1 state k=1 for BOTH57M. This contradiction should be fixed.","section":"Sec. 4.3"},{"comment":"The ablation results are reported without variance or multiple seeds. Given the large claimed improvements, at least seed-level standard deviations would increase confidence that the differences are not noise.","section":"Table 6"},{"comment":"For BOTH57M, the paper says T2MT and MotionGPT baselines were trained from source code with 'recommended hyper-parameters,' but it does not state whether these hyper-parameters were tuned on the BOTH57M validation split. Please specify.","section":"Sec. 4.1"}],"recommendation":"major_revision","confidential_remarks":"The reported in-distribution numbers are plausible and the ablation design is reasonable, but the paper's central claim about external text sources is not tested in the experiments. The authors should either add the missing external-text and rare-motion experiments or substantially reframe the contributions. I do not see a fatal flaw in the method itself, but the current framing overstates what is demonstrated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is real: instead of mapping motion directly to a high-level caption, HiCAM2T first generates a low-level joint description with an LLM-tuned decoder, then retrieves candidate high-level captions by embedding similarity and fuses them with motion features for the final output. That two-step framing is not in the cited prior work, and the abalations show each piece earns its keep. The empirical tables are also consistent: HiCAM2T beats T2MT, M2T-INT, MotionGPT, LaMP, and MotionLLM on KIT, HumanML3D, and BOTH57M, under both lemmatized and non-lemmatized evaluation, with gains that look substantial, especially on BOTH57M. The authors also report ablations for k and for the retrieval step, which strengthens the case that the mechanism, not just the larger model, is doing the work.\n\nThe soft spots are real but not fatal. The biggest one: the abstract promises \"external text sources\" and \"movements not covered in existing motion-text datasets,\" but no experiment delivers that. The only database enrichment is adding validation captions from the same datasets, not standalone external text, and there is no evaluation on rare or held-out action categories. So the headline generalization is unverified. The low-level descriptions are generated by Falcon 40B from the very high-level captions they later help predict, and fewer than 50 of tens of thousands were manually checked. That is a legitimate weakness, though the authors acknowledge it and it does not invalidate the in-distribution results. Also note that while the retrieval database at test time does not contain the test captions, the low-level labels are derived from the high-level labels, so the intermediate supervision is not independent of the target. No code or data artifacts are provided, which hurts reproducibility but is not a scientific flaw per se.\n\nI agree with the reader's conditional verdict. The paper deserves a serious referee, but the referee should push for an honest test of the external-text claim: add captions from a truly separate text source, evaluate on unseen motion categories, and measure retrieval quality. If that comes back positive, this could be a useful contribution to an applied subfield. If not, the method still works as a strong within-dataset retrieval augmentation, and the paper should be repositioned accordingly.","headline":"A genuinely new two-step motion captioning pipeline with strong benchmark results, but the paper's marquee claim about external text sources is never actually tested.","tokens_in":14288,"tokens_out":1567,"would_cite":false,"duration_ms":18887,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that motion captioning improves by first generating a low-level description of the movement and then retrieving high-level captions from an expandable text database, outperforming direct motion-to-caption models.","keywords":["motion captioning","motion-to-text","hierarchical generation","retrieval-augmented captioning","LLM-generated descriptions","cosine similarity retrieval","human motion datasets"],"falsifier":"Take a held-out set of motions with human-written low-level descriptions. If, when human descriptions replace the LLM-generated ones in retrieval, the final caption metrics do not improve or correlate with retrieval precision, the unverified synthetic labels are the weak link. Equivalently, measure precision@k of retrieving the correct high-level caption from a database where the query is a low-level description: if same-action low-level descriptions are not consistently more cosine-similar than different-action ones, the retrieval bridge fails.","tokens_in":13459,"feed_emoji":"🏃","tokens_out":7958,"duration_ms":77024,"temperature":0.7,"pith_summary":"This paper claims that motion-to-text models are held back by trying to jump straight from raw motion to a high-level caption such as \"a person doing jumping jacks.\" The authors propose a two-step pipeline, HiCAM2T: first a vision-transformer encoder and language decoder write a detailed low-level description of the movement; then that description is matched by cosine similarity against a database of (low-level description, high-level caption) pairs, and the top matches are fed together with the motion features to a text decoder that produces the final caption. The database can be expanded with new text-only entries after training, so the system can absorb high-level captions it never saw. On HumanML3D, KIT, and BOTH57M, the paper reports average improvements of 6% to 50% over the M2T-Interpretable baseline across BLEU-1, BLEU-4, CIDEr, and ROUGE-L, with larger gains in settings where captions are sparse or vocabulary is large. The motivating point is that captioning accuracy need not wait for expensive new motion-capture data.","feed_headline":"6-50% better motion captions without new motion data","feed_subtitle":"The method writes a detailed description first, then retrieves high-level captions from an expandable text database.","key_machinery":"The load-bearing mechanism is the retrieval bridge between two text decoders. A Vision Transformer treats the motion as an image and extracts features; the first text decoder (Distilled-GPT2 or GPT2) turns those features into a low-level motion description; Sentence-BERT embeds that description; cosine similarity against stored embeddings of low-level descriptions selects the top k high-level captions; a second decoding pass concatenates the motion features with the retrieved captions to generate the final high-level caption. The database entries are triples (motion, high-level caption, low-level description plus embedding), and the whole system is trained with a weighted sum of three losses","core_discovery":"The central claim is that a direct mapping from motion to high-level caption is the wrong bottleneck. HiCAM2T instead treats captioning as two linked tasks: first translate the motion into a low-level explanation of what the body is doing (arms, legs, torso, timing), then use that explanation as a query to retrieve candidate high-level captions from a database that can include text-only sources with no paired motion. The low-level descriptions are generated offline by an LLM from the existing high-level captions, converting every motion-text pair into a motion-and-two-captions triple. At inference, the motion encoder produces features, the first text decoder writes the low-level description,","pith_inferences":["The paper's strongest claim about external text sources is not directly tested: the database-enrichment experiments add captions from the validation split, not from genuinely outside corpora, so a controlled test with true external text remains open.","The entire pipeline depends on the faithfulness of LLM-generated low-level descriptions, yet fewer than 50 of tens of thousands were verified; a small human-verified evaluation set would directly measure this dependency.","The retrieval step assumes Sentence-BERT cosine similarity tracks action identity, an assumption the paper never validates; a labeled similarity benchmark could test it.","If the hierarchy is sound, the same low-level-then-retrieve design could transfer to other modalities, such as video or sensor streams, where high-level labels are scarce but detailed descriptions can be synthesized."],"forward_implications":["If the reported gains hold, motion captioning can improve without collecting new motion-caption pairs: text-only sources can be folded into the retrieval database, and new high-level captions can be added at inference time at linear retrieval cost.","Rare motions and large-vocabulary datasets benefit most, because the decoder no longer has to memorize vocabulary in its weights; the knowledge sits in the database.","The two-stage design decouples 'what is the body doing' from 'what do we call it', a natural decomposition that can generalize to captioning in other time-series domains.","The 1-13% gains from the enriched-database variant show that expanding the retrieval database alone, without retraining, is a viable way to adapt a captioning system.","Because the system outputs a low-level description as an intermediate artifact, it also yields granular movement descriptions usable for applications beyond final captions."],"supporting_citations":[{"why":"Falcon 40B LLM generates all low-level descriptions from high-level captions, producing the training targets and retrieval entries.","marker":"(Almazrouei et al., 2023)"},{"why":"Sentence-BERT supplies the text encoder whose embeddings are compared by cosine similarity for retrieval and contrastive training.","marker":"(Reimers & Gurevych, 2019)"},{"why":"M2T-Interpretable is the state-of-the-art baseline the paper compares against and reports 6-50% average gains over.","marker":"(Radouane et al., 2023)"},{"why":"HumanML3D provides paired motion-caption data and the train/validation/test split protocol used in evaluation.","marker":"(Guo et al., 2022a)"},{"why":"KIT motion-language dataset motivates the method via its sparse annotations, e.g., 'jumping jacks' appears only once.","marker":"(Plappert et al., 2016)"},{"why":"BOTH57M provides the hand-joint motion benchmark where the method reports its largest relative gains.","marker":"(Zhang et al., 2024)"},{"why":"Vision Transformer serves as the motion encoder, treating the motion matrix as an image.","marker":"(Dosovitskiy et al., 2020)"},{"why":"Distilled-GPT2 is the text decoder chosen for KIT and BOTH57M in the hierarchical pipeline.","marker":"(Sanh et al., 2019)"},{"why":"GPT-2 is the text decoder chosen for HumanML3D in the hierarchical pipeline.","marker":"(Radford et al., 2019)"}],"fun_headline_variants":["Hierarchical LLM + retrieval boosts motion caption accuracy up to 50%","Motion captions see 6-50% lift via two-step LLM and text retrieval","External text data teaches AI to caption unseen motion better","Two-step captioning: LLM writes details, retrieval finds high-level","No new motion data needed: hierarchical method improves captions up to 50%"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The method assumes that the large-language-model-generated low-level descriptions are faithful enough to train the motion encoder and to act as reliable retrieval keys, even though fewer than 50 of the tens of thousands of descriptions were human-verified.","fun_headline_variants_meta":{"raw":{"variants":["Hierarchical LLM + retrieval boosts motion caption accuracy up to 50%","Motion captions see 6-50% lift via two-step LLM and text retrieval","External text data teaches AI to caption unseen motion better","Two-step captioning: LLM writes details, retrieval finds high-level","No new motion data needed: hierarchical method improves captions up to 50%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000251,"raw_usage":{"total_tokens":1428,"prompt_tokens":815,"completion_tokens":613,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":514}},"tokens_in":559,"tokens_out":613,"duration_ms":7344,"temperature":1.0,"reasoning_tokens":514,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T12:29:16.157251+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of motions with human-written low-level descriptions. If, when human descriptions replace the LLM-generated ones in retrieval, the final caption metrics do not improve or correlate with retrieval precision, the unverified synthetic labels are the weak link. Equivalently, measure precision@k of retrieving the correct high-level caption from a database where the query is a low-level description: if same-action low-level descriptions are not consistently more cosine-similar than different-action ones, the retrieval bridge fails.","supporting_citations":[],"review_version":1}